[benchmark] Make paranoid mode configurable #15900

georgemitenkov · 2025-02-06T09:31:52Z

Description

Adds a flag to disable paranoid mode for benchmarking. This way we can measure performance with and without on historical workloads.

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2025-02-06T09:31:56Z

⏱️ 49m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
check-dynamic-deps	17m	🟩 🟩 🟩 🟩 🟩 (+2 more)
rust-cargo-deny	9m	🟩 🟩 🟩 ⬜ 🟩
rust-doc-tests	6m	🟩
test-target-determinator	5m	🟩
execution-performance / test-target-determinator	5m	🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩 (+2 more)
general-lints	2m	🟩 🟩 🟩 🟩 🟩
fetch-last-released-docker-image-tag	2m	🟩
file_change_determinator	57s	🟩 ⬜ 🟩 🟩 🟩 (+1 more)
permission-check	18s	🟩 🟩 🟩 🟩 🟩 (+2 more)
permission-check	17s	🟩 🟩 🟩 🟩 🟩 (+2 more)
file_change_determinator	11s	🟩
execution-performance / single-node-performance	9s	🟩
permission-check	4s	🟩
determine-docker-build-metadata	2s	🟩

🚨 2 jobs on the last run were significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-cargo-deny	3m	2m
execution-performance / single-node-performance	9s	20m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

georgemitenkov · 2025-02-06T09:32:10Z

[benchmark] Make gas metering conditional #15901
[benchmark] Make paranoid mode configurable #15900 👈 (View in Graphite)
[benchmark] Add option for package overrides #15841
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

graphite-app · 2025-02-06T09:34:27Z

aptos-move/replay-benchmark/README.md

+for the first `N` blocks (the blocks will still be executed as a "warm-up").
+
+Execution can also be configured. By using `--disable-paranoid-mode`, the Move VM will not use
+runtime type checks, possible making execution faster.


Typo: "possible" should be "possibly" in "possible making execution faster"

Spotted by Graphite Reviewer

Is this helpful? React 👍 or 👎 to let us know.

Fixed in #15901

gelash · 2025-02-10T19:33:54Z

aptos-move/replay-benchmark/src/commands/initialize.rs

+        value_delimiter = ' ',
+        help = "List of space-separated paths to compiled / built packages with Move code"
+    )]
+    override_packages: Vec<String>,


wasn't this in a different PR?

Need to rebase the branch, the other PR has just landed

github-actions · 2025-02-10T20:54:12Z

✅ Forge suite `compat` success on `255673e8f2757ed8634a7d65d7aca11de7016cde` ==> `4957ba68468c8f55ed92bd32e50cb0eb52c9b31e`

Compatibility test results for 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e (PR)
1. Check liveness of validators at old version: 255673e8f2757ed8634a7d65d7aca11de7016cde
compatibility::simple-validator-upgrade::liveness-check : committed: 11584.70 txn/s, latency: 2715.44 ms, (p50: 2600 ms, p70: 2800, p90: 3800 ms, p99: 5000 ms), latency samples: 387560
2. Upgrading first Validator to new version: 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 3436.09 txn/s, latency: 8789.85 ms, (p50: 9500 ms, p70: 10100, p90: 10600 ms, p99: 10800 ms), latency samples: 78900
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 3351.28 txn/s, latency: 10293.19 ms, (p50: 11300 ms, p70: 11500, p90: 11700 ms, p99: 11800 ms), latency samples: 127960
3. Upgrading rest of first batch to new version: 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 3312.62 txn/s, latency: 8928.56 ms, (p50: 9500 ms, p70: 10800, p90: 11600 ms, p99: 11800 ms), latency samples: 72520
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3341.06 txn/s, latency: 10348.44 ms, (p50: 11300 ms, p70: 11600, p90: 11700 ms, p99: 11900 ms), latency samples: 127720
4. upgrading second batch to new version: 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 5987.13 txn/s, latency: 5066.27 ms, (p50: 5800 ms, p70: 6200, p90: 6700 ms, p99: 6800 ms), latency samples: 114340
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 5949.16 txn/s, latency: 5736.76 ms, (p50: 6000 ms, p70: 6500, p90: 6900 ms, p99: 7000 ms), latency samples: 204400
5. check swarm health
Compatibility test for 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e passed
Test Ok

github-actions · 2025-02-10T20:56:11Z

✅ Forge suite `realistic_env_max_load` success on `4957ba68468c8f55ed92bd32e50cb0eb52c9b31e`

two traffics test: inner traffic : committed: 12364.86 txn/s, submitted: 12366.75 txn/s, expired: 1.89 txn/s, latency: 3197.62 ms, (p50: 2700 ms, p70: 3000, p90: 3600 ms, p99: 10400 ms), latency samples: 4701420
two traffics test : committed: 99.97 txn/s, latency: 1763.33 ms, (p50: 1600 ms, p70: 1700, p90: 2300 ms, p99: 3300 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.783, avg: 1.544", "ConsensusProposalToOrdered: max: 0.334, avg: 0.327", "ConsensusOrderedToCommit: max: 0.725, avg: 0.557", "ConsensusProposalToCommit: max: 1.052, avg: 0.884"]
Max non-epoch-change gap was: 1 rounds at version 22987 (avg 0.00) [limit 4], 1.85s no progress at version 22987 (avg 0.22s) [limit 15].
Max epoch-change gap was: 1 rounds at version 5071088 (avg 0.50) [limit 4], 2.41s no progress at version 5071088 (avg 1.60s) [limit 16].
Test Ok

georgemitenkov mentioned this pull request Feb 6, 2025

[benchmark] Add option for package overrides #15841

Merged

8 tasks

georgemitenkov requested review from vineethk, runtian-zhou, vgao1996, wrwg, gelash and igor-aptos February 6, 2025 09:32

georgemitenkov marked this pull request as ready for review February 6, 2025 09:33

graphite-app bot reviewed Feb 6, 2025

View reviewed changes

georgemitenkov mentioned this pull request Feb 6, 2025

[benchmark] Make gas metering conditional #15901

Open

9 tasks

msmouse approved these changes Feb 6, 2025

View reviewed changes

vgao1996 approved these changes Feb 6, 2025

View reviewed changes

georgemitenkov force-pushed the george/replay-benchmark-package-override branch from 5e3e076 to c8171a9 Compare February 8, 2025 16:55

georgemitenkov force-pushed the george/replay-benchmark-overrides branch 2 times, most recently from efc8134 to e7c022d Compare February 10, 2025 16:44

Base automatically changed from george/replay-benchmark-package-override to main February 10, 2025 18:38

gelash reviewed Feb 10, 2025

View reviewed changes

[benchmark] Make paranoid mode configurable

4957ba6

georgemitenkov force-pushed the george/replay-benchmark-overrides branch from e7c022d to 4957ba6 Compare February 10, 2025 20:27

georgemitenkov enabled auto-merge (squash) February 10, 2025 20:28

This comment has been minimized.

Sign in to view

georgemitenkov merged commit 9fee3cd into main Feb 10, 2025
84 of 88 checks passed

georgemitenkov deleted the george/replay-benchmark-overrides branch February 10, 2025 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[benchmark] Make paranoid mode configurable #15900

[benchmark] Make paranoid mode configurable #15900

georgemitenkov commented Feb 6, 2025 •

edited

Loading

trunk-io bot commented Feb 6, 2025 •

edited

Loading

georgemitenkov commented Feb 6, 2025 •

edited

Loading

graphite-app bot Feb 6, 2025

georgemitenkov Feb 6, 2025

gelash Feb 10, 2025

georgemitenkov Feb 10, 2025

georgemitenkov Feb 10, 2025

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 10, 2025

github-actions bot commented Feb 10, 2025

[benchmark] Make paranoid mode configurable #15900

[benchmark] Make paranoid mode configurable #15900

Conversation

georgemitenkov commented Feb 6, 2025 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Feb 6, 2025 • edited Loading

georgemitenkov commented Feb 6, 2025 • edited Loading

graphite-app bot Feb 6, 2025

Choose a reason for hiding this comment

georgemitenkov Feb 6, 2025

Choose a reason for hiding this comment

gelash Feb 10, 2025

Choose a reason for hiding this comment

georgemitenkov Feb 10, 2025

Choose a reason for hiding this comment

georgemitenkov Feb 10, 2025

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 10, 2025

✅ Forge suite compat success on 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e

github-actions bot commented Feb 10, 2025

✅ Forge suite realistic_env_max_load success on 4957ba68468c8f55ed92bd32e50cb0eb52c9b31e

georgemitenkov commented Feb 6, 2025 •

edited

Loading

trunk-io bot commented Feb 6, 2025 •

edited

Loading

georgemitenkov commented Feb 6, 2025 •

edited

Loading

✅ Forge suite `compat` success on `255673e8f2757ed8634a7d65d7aca11de7016cde` ==> `4957ba68468c8f55ed92bd32e50cb0eb52c9b31e`

✅ Forge suite `realistic_env_max_load` success on `4957ba68468c8f55ed92bd32e50cb0eb52c9b31e`