Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Criterium 0.5.x #45

Open
wants to merge 90 commits into
base: develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
e033491
Merge branch 'release/0.4.3'
hugoduncan Jan 22, 2014
d11522d
Merge branch 'release/0.4.4'
hugoduncan Feb 16, 2016
50a52b6
Merge branch 'release/0.4.5'
hugoduncan Apr 19, 2019
4838e8d
Merge branch 'release/0.4.6'
hugoduncan Jul 3, 2020
01fae92
feat: 0.5.0-dev core
hugoduncan Dec 18, 2021
e25f083
chore: update dependencies to latest
hugoduncan Dec 28, 2022
9fdaceb
chore: update clojure-lsp version
hugoduncan Dec 28, 2022
d5d5784
chore: fix lint issues
hugoduncan Dec 28, 2022
ce75e2e
prep for release
hugoduncan Dec 20, 2024
d990db2
docs: improve criterium.bench docstrings
hugoduncan Dec 20, 2024
fb0d62a
refactor: move last-bench* to implementation namespace
hugoduncan Dec 20, 2024
93b251f
refactor: remove unused exclusion of core time
hugoduncan Dec 20, 2024
0b902f4
fix: remove portal padding and require by default
hugoduncan Dec 20, 2024
c23cc7f
feat: add a :collect-plan :without-jit-warmup option
hugoduncan Dec 20, 2024
23f516b
docs(measured): improve documentation clarity
hugoduncan Dec 20, 2024
132115c
docs(measured): add note on non-primitive arg
hugoduncan Dec 20, 2024
0cd24ac
docs(analyse): improve documentation
hugoduncan Dec 20, 2024
e9745b0
refactor(benchmark): improve naming and doc
hugoduncan Dec 20, 2024
8e6b288
docs(metric): improve documentation
hugoduncan Dec 20, 2024
60dc116
docs: add problem statement and glossary
hugoduncan Dec 20, 2024
f877e16
chore: add have assertion macro
hugoduncan Dec 21, 2024
70d9d2b
chore: remove use of metadata, improve data maps
hugoduncan Dec 21, 2024
497282f
add portal submit fn, and headers
hugoduncan Dec 26, 2024
60fd087
fixup platform
hugoduncan Dec 27, 2024
e8f3d31
chore: add jvm.impl tests
hugoduncan Dec 28, 2024
4094e59
chore: add jvm doc strings and tests
hugoduncan Dec 29, 2024
d5bbedd
feat: :viewer :none
hugoduncan Dec 29, 2024
347f550
chore: platform test
hugoduncan Dec 29, 2024
851487c
chore: improve instrument documentation
hugoduncan Dec 29, 2024
90d73ee
feat: make instrument! idempotent
hugoduncan Dec 29, 2024
aed7f34
fix: add instrument tests and fix
hugoduncan Dec 29, 2024
296ee45
feat: add instrument-fn
hugoduncan Dec 29, 2024
93d5b6d
fix: add trigger docs, tests and fix
hugoduncan Dec 29, 2024
7b84d48
chore: factor out Sampler protocol
hugoduncan Dec 29, 2024
20bc43d
chore: change trigger to use Sampler protocol
hugoduncan Dec 29, 2024
d8c7078
chore: remove boxing from measured-test
hugoduncan Dec 30, 2024
36ce6d3
chore: document agent ns and functions, add tests
hugoduncan Dec 30, 2024
032f076
chore: clean up boxed warnings
hugoduncan Dec 30, 2024
8ddacf9
fix build
hugoduncan Dec 30, 2024
86df092
chore: rename metrics-configs to metrics-defs
hugoduncan Dec 31, 2024
3ec02a3
chore: add approx= test assertion
hugoduncan Dec 31, 2024
a710e2d
add histogram to print and pprint
hugoduncan Dec 31, 2024
f89f2c3
chore: move test-data namespace
hugoduncan Dec 31, 2024
45a9348
chore: move batch-transforms function
hugoduncan Dec 31, 2024
7c26416
chore: make transform use more consistent
hugoduncan Dec 31, 2024
3421ad9
fix: improve sampling batch size calculation
hugoduncan Dec 31, 2024
698ce7b
chore: automatically add vega lite schema key
hugoduncan Dec 31, 2024
2339daf
fix: make portal/flush wait for all output
hugoduncan Dec 31, 2024
6c6fc1b
fix: time dimension scaling
hugoduncan Dec 31, 2024
cebbc0b
fix: make tables contain numbers (not strings)
hugoduncan Dec 31, 2024
2a7622d
fix: clean up pprint and print histogram output
hugoduncan Jan 1, 2025
e4ac14d
chore: use collector rather than pipeline
hugoduncan Jan 2, 2025
7350bac
refactor(benchmark): Split benchmark composition into separate analys…
hugoduncan Jan 2, 2025
e869d69
chore: improve bench-measured* argument clarity
hugoduncan Jan 2, 2025
d1f1b6d
feat: simplified config and plans
hugoduncan Jan 2, 2025
cf545e5
fix: handle multiple metrics
hugoduncan Jan 3, 2025
4e652ff
chore: separate test and kaocha alisases
hugoduncan Jan 12, 2025
d31b075
chore: add t-digest
hugoduncan Jan 3, 2025
a1fb6aa
feat: add sampled-fn
hugoduncan Jan 12, 2025
625ab67
chore: move some files to reduce context size
hugoduncan Jan 12, 2025
4249289
chore: avoid boxing when passed :time-fn
hugoduncan Jan 12, 2025
612f360
factor out types namespace
hugoduncan Jan 12, 2025
2d6afad
chore: test aliases updates
hugoduncan Jan 12, 2025
551f5f0
chore: remove some boxing in tests
hugoduncan Jan 12, 2025
05007ec
fix: bootstrap
hugoduncan Jan 12, 2025
aaba35a
chore: improve metric-config?
hugoduncan Jan 12, 2025
9776604
chore: fix print-test
hugoduncan Jan 12, 2025
987c997
chore: fix sampled-stats-test
hugoduncan Jan 13, 2025
e459652
chore: fix portal-test
hugoduncan Jan 13, 2025
892ee6a
fix pprint, platform, and more tests
hugoduncan Jan 15, 2025
f527e5b
chore: fix platform
hugoduncan Jan 15, 2025
dbb6092
chore: fix instrument test
hugoduncan Jan 15, 2025
005b305
fix config-test
hugoduncan Jan 15, 2025
01c7127
fix analyse-test
hugoduncan Jan 15, 2025
28fdc68
fix trigger-test
hugoduncan Jan 18, 2025
41c86bb
fix metric-test
hugoduncan Jan 18, 2025
36cbcf7
fix metrics-test
hugoduncan Jan 18, 2025
a95ad3d
fix collect-plan-test
hugoduncan Jan 18, 2025
2f82b20
fix core
hugoduncan Jan 18, 2025
69466ac
fix config unknown-keys check
hugoduncan Jan 19, 2025
90d042c
fix port-test
hugoduncan Jan 19, 2025
d0c3748
chore: add sampled-fn test
hugoduncan Jan 19, 2025
51c22d5
fix: handle potential division by zero in merging-digest
hugoduncan Jan 22, 2025
e7069d3
chore: fix some merging digest tests
hugoduncan Jan 25, 2025
e8fb879
feat: merging digest transform and stats
hugoduncan Jan 25, 2025
28910d7
chore: add cond*
hugoduncan Jan 25, 2025
eff10d5
chore: fix t-digest tests and improve impl
hugoduncan Jan 25, 2025
c0fb202
chore: add merging-digest transform tests
hugoduncan Jan 25, 2025
74ada72
feat: implement transform-log for t-digest samples
hugoduncan Jan 25, 2025
0935c42
feat: add more analysis on digests
hugoduncan Jan 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix pprint, platform, and more tests
  • Loading branch information
hugoduncan committed Jan 15, 2025
commit 892ee6a769d5a1e70f19365b0e3c38c8105fcabc
2 changes: 1 addition & 1 deletion bases/criterium/src/criterium/bench.clj
Original file line number Diff line number Diff line change
@@ -82,7 +82,7 @@
(defn- return-value
"Extract the returned value for the sampled."
[config bench-map]
(get-in bench-map (-> config :return-value)))
(get-in bench-map (:return-value config)))

(defn bench-measured
"Evaluate measured and output the benchmark time.
4 changes: 3 additions & 1 deletion bases/criterium/src/criterium/bench/config.clj
Original file line number Diff line number Diff line change
@@ -83,7 +83,9 @@
:collect-plan collect-plan
:collector-config collector-config
:viewer (:viewer options-map :print)
:return-value (:return-value options-map [:expr-value]))
:return-value (:return-value
options-map
[:samples :expr-value]))

(= scheme-type :with-jit-warmup)
(assoc :analyse (or analyse
3 changes: 1 addition & 2 deletions bases/criterium/src/criterium/collect_plan.clj
Original file line number Diff line number Diff line change
@@ -164,5 +164,4 @@
"Collect metrics from the measured according to the collect-plan.
Return a results-map."
[collect-plan collector measured]
(-> (impl/collect* collect-plan collector measured)
(vary-meta assoc :type :criterium/sampled)))
(impl/collect* collect-plan collector measured))
39 changes: 22 additions & 17 deletions bases/criterium/src/criterium/platform.clj
Original file line number Diff line number Diff line change
@@ -4,19 +4,19 @@
[clojure.pprint :as pp]
[criterium.analyse]
[criterium.bench :as bench]
[criterium.benchmark :as benchmark]
[criterium.collect-plan.config :as collect-plan-config]
[criterium.collector :as collector]
[criterium.jvm :as jvm]
[criterium.measured :as measured])
(:gen-class))

(def ^:private benchmark
(benchmark/->benchmark
{:analyse [:transform-log
[:quantiles {:quantiles [0.9 0.99 0.99]}]
:outliers
[:stats {:samples-id :log-samples}]
:event-stats]}))
{:analyse [:transform-log
[:quantiles {:quantiles [0.9 0.99 0.99]}]
:outliers
[:stats {:samples-id :log-samples}]
:event-stats]
:view []})

;;; nanoTime latency
(def ^:private timestamp-measured
@@ -28,16 +28,21 @@
([] (nanotime-latency {}))
([options]
(bench/bench-measured
timestamp-measured
(merge
{:collect-plan {:scheme-type :with-jit-warmup
:batch-time-ns 100000
:num-measure-samples 1000}
:limit-time-s 20
:benchmark benchmark
:viewer :none
:return-value [::nil]}
options))))
benchmark
{:collect-plan
(collect-plan-config/collect-plan-config
:with-jit-warmup
{:batch-time-ns 100000
:num-measure-samples 1000
:limit-time-ns 20000000000})
:collector-config
{:stages [:measured-args :compilation :garbage-collector],
:terminator :elapsed-time}
:return-value [::nil]
:viewer :none}
options)
timestamp-measured)))

;;; nanoTime granularity

@@ -172,7 +177,7 @@
([options]
(let [options (merge
options
{:return-value [:data :stats :stats]})]
{:return-value [:stats :stats]})]
[(assoc (nanotime-latency options) :name "latency")
(assoc (nanotime-granularity options) :name "granularity")
(assoc (constant-long options) :name "constant-long")
2 changes: 1 addition & 1 deletion bases/criterium/src/criterium/viewer/pprint.clj
Original file line number Diff line number Diff line change
@@ -85,7 +85,7 @@
(get-in outlier-sig (:path m))))))

(defmethod view/outlier-significance* :pprint
[view data-map]
[_ view data-map]
(print-outlier-significances view data-map))

(defn- flatten-events [sample metrics-defs index]
50 changes: 24 additions & 26 deletions bases/criterium/test/criterium/viewer/pprint_test.clj
Original file line number Diff line number Diff line change
@@ -12,15 +12,15 @@

(def expected-stats-1
[""
"| :metric | :mean-minus-3sigma | :mean | :mean-plus-3sigma | :min-val | :max-val |"
"|--------------+--------------------+--------+-------------------+----------+----------|"
"| Elapsed Time | 88.0 ns | 100 ns | 112 ns | 89.0 ns | 114 ns |"])
"| :_metric | :mean-minus-3sigma | :mean | :mean-plus-3sigma | :min-val | :max-val |"
"|-----------------+--------------------+-------+-------------------+----------+----------|"
"| Elapsed Time ns | 88.0 | 100.0 | 112.0 | 89.0 | 114.0 |"])

(def expected-stats-2
[""
"| :metric | :mean-minus-3sigma | :mean | :mean-plus-3sigma | :min-val | :max-val |"
"|--------------+--------------------+---------+-------------------+----------+----------|"
"| Elapsed Time | 1.00 ns | 1.00 ns | 1.00 ns | 1.00 ns | 1.00 ns |"])
"| :_metric | :mean-minus-3sigma | :mean | :mean-plus-3sigma | :min-val | :max-val |"
"|-----------------+--------------------+-------+-------------------+----------+----------|"
"| Elapsed Time ns | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |"])

(deftest pprint-stats-test
(testing "print-stats"
@@ -29,24 +29,23 @@
(trimmed-lines
(with-out-str
(view/stats*
:pprint
{}
(-> (test-data/bench-stats-map)
(assoc :viewer :pprint)))))))
(:data (test-data/bench-stats-map)))))))

(is (= expected-stats-2
(let [bench-map (-> (test-data/samples-with-2-values-map)
(assoc :viewer :pprint))
(let [data-map (:data (test-data/samples-with-2-values-map))
stats (analyse/stats)
view-stats (view/stats)]
(trimmed-lines
(with-out-str
(-> bench-map
stats
view-stats)))))))))
(->> data-map
stats
(view-stats :pprint))))))))))

(def expected-counts
[""
"| :metric | :low-severe | :low-mild | :high-mild | :high-severe |"
"| :_metric | :low-severe | :low-mild | :high-mild | :high-severe |"
"|--------------+-------------+-----------+------------+--------------|"
"| Elapsed Time | 0 | 2 | 3 | 0 |"])

@@ -58,8 +57,8 @@
(trimmed-lines
(with-out-str
((view/outlier-counts)
(-> (test-data/outlier-count-map)
(assoc :viewer :pprint))))))))))
:pprint
(:data (test-data/outlier-count-map))))))))))

(deftest print-outlier-significance-test
(testing "print-outlier-significance"
@@ -71,27 +70,26 @@
(trimmed-lines
(with-out-str
((view/outlier-significance)
(-> (test-data/outlier-significance-map)
(assoc :viewer :pprint))))))))))
:pprint
(:data (test-data/outlier-significance-map))))))))))

(def ^:private expected-event-stats
[""
"| :metric | :sample-count | :loaded-count | :unloaded-count | :time-ms | :total-sample-count | :total-count | :total-time-ms |"
"|-------------------+---------------+---------------+-----------------+----------+---------------------+--------------+----------------|"
"| ClassLoader | 1 | 1 | 1 | | | | |"
"| JIT compilation | 1 | | | 3.00 ms | | | |"
"| Garbage Collector | | | | | 1 | 2 | 1.00 ms |"])
"| ClassLoader | 1.0 | 1.0 | 1.0 | | | | |"
"| JIT compilation | 1.0 | | | 3.00 ms | | | |"
"| Garbage Collector | | | | | 1.0 | 2.0 | 1.00 ms |"])

(deftest pprint-event-stats-test
(testing "print-event-stats"
(testing "prints via report"
(is (= expected-event-stats
(let [bench-map (-> (test-data/samples-for-event-stats-map)
(assoc :viewer :pprint))
(let [data-map (:data (test-data/samples-for-event-stats-map))
event-stats (analyse/event-stats)
view (view/event-stats)]
(trimmed-lines
(with-out-str
(-> bench-map
event-stats
view)))))))))
(->> data-map
event-stats
(view :pprint))))))))))
2 changes: 1 addition & 1 deletion deps.edn
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@
org.clojure/test.check {:mvn/version "1.1.1"}}
:jvm-opts ["-XX:-OmitStackTraceInFastThrow"
"-Dclojure.main.report=stderr"
"-agentpath:agent-cpp/libcriterium.dylib"]}
#_"-agentpath:agent-cpp/libcriterium.dylib"]}
:kaocha {:extra-deps
{lambdaisland/kaocha {:mvn/version "1.91.1392"}
lambdaisland/kaocha-cloverage {:mvn/version "1.1.89"}