[benchmark] Add option for package overrides #15841

georgemitenkov · 2025-01-29T14:29:40Z

Description

This PR adds support for package overriding: one can specify --override-packages P1 P2 P3 and replay historical transactions with new code (e.g., framework). This allows to check performance (benchmark option) or gas usage (diff option).

Note: one needs to be careful overriding packages which add new package dependencies. For example, it is possible that stdlib adds a module, which is used in framework code. For override, one needs to provide both packages.

How Has This Been Tested?

Key Areas to Review

Type of Change

New feature

Which Components or Systems Does This Change Impact?

Other (specify)

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2025-01-29T14:29:44Z

⏱️ 27m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
check-dynamic-deps	14m	🟩 🟩 🟩 🟩 🟩
rust-cargo-deny	8m	🟩 ⬜ 🟩 🟩 🟩
general-lints	2m	🟩 ⬜ 🟩 🟩 🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩
permission-check	17s	🟩 🟩 🟩 🟩 🟩
permission-check	15s	🟩 🟩 🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

georgemitenkov · 2025-01-29T14:30:02Z

[benchmark] Make gas metering conditional #15901
[benchmark] Make paranoid mode configurable #15900
[benchmark] Add option for package overrides #15841 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

vineethk

Looks great and a very useful functionality. Left a few minor comments.

aptos-move/replay-benchmark/src/overrides.rs

aptos-move/replay-benchmark/src/commands/initialize.rs

vineethk · 2025-01-29T16:05:09Z

aptos-move/replay-benchmark/src/commands/initialize.rs

+        long,
+        num_args = 1..,
+        value_delimiter = ' ',
+        help = "List of space-separated paths to compiled / built packages with Move code"


Another cool feature this brings (and I think we should be explicit about it in the readme) is that you can compile using different compiler versions/optimization levels/experimental compiler stuff and just point to the built packages, giving the ability to compare them.

Yes yes yes! Exactly, but for that I need to modify CLI to pass flags to build option, or instead get already build packages from the path (ideally)

vineethk · 2025-01-29T16:07:15Z

aptos-move/replay-benchmark/src/commands/initialize.rs

+        //   1. Override gas schedule, to track the costs of charging gas or tracking limits.
+        //   2. BlockExecutorConfigFromOnchain to experiment with different block cutting based
+        //      on gas limits?.
+        //   3. Build options for package overrides.


Ah, so this would allow pointing to just source. Perhaps mention that right now, the build option is only used for getting the bytecode version (and that this wont recompile the package with move-2).

Right now compilation happens with hardcoded options, ideally one should be able to paramterize this, do you have specific things in mind I should add? e.g., how do I enable optimization levels, etc.

My original plan was to point to already build package (so that using compiler is external), but I did not find a way to retrieve the package from the filesystem (i.e., we don't have APIs for that, etc.) so instead just used this for now. You can probably suggest more here from compiler side?

aptos-move/replay-benchmark/src/commands/initialize.rs

aptos-move/replay-benchmark/src/overrides.rs

gelash · 2025-02-06T18:34:34Z

aptos-move/replay-benchmark/src/overrides.rs

+                                package_address, err
+                            )
+                        })
+                        .expect("Package registry for override must always exist"),


Is there an easy workflow to emulate publishing a new package and using that elsewhere? (but i guess for that purpose, can always include the new one inside the updated package)?

Not that I know of, package system was built on top of the modules so it is quite hacky

aptos-move/replay-benchmark/src/overrides.rs

gelash · 2025-02-06T18:56:49Z

aptos-move/replay-benchmark/src/overrides.rs

+                    let mut metadata_idx = None;
+                    for (idx, package_metadata) in package_registry.packages.iter().enumerate() {
+                        if package_metadata.name == metadata.name {
+                            metadata_idx = Some(idx);


we can assign it here and only handle None below?

I am not sure, you need to clone metadata then (Rust complains)? Even though it is clearly that metadata cannot be moved if we fallback to break.

github-actions · 2025-02-10T17:14:41Z

✅ Forge suite `realistic_env_max_load` success on `304ce07afe7115510b88ca8c136d8ae909d12d14`

two traffics test: inner traffic : committed: 12814.41 txn/s, submitted: 12817.15 txn/s, expired: 2.74 txn/s, latency: 3097.79 ms, (p50: 3000 ms, p70: 3000, p90: 3600 ms, p99: 4800 ms), latency samples: 4872580
two traffics test : committed: 99.99 txn/s, latency: 2256.31 ms, (p50: 2200 ms, p70: 2600, p90: 3100 ms, p99: 4000 ms), latency samples: 2000
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.321, avg: 0.842", "ConsensusProposalToOrdered: max: 0.324, avg: 0.317", "ConsensusOrderedToCommit: max: 0.944, avg: 0.642", "ConsensusProposalToCommit: max: 1.262, avg: 0.959"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 2.16s no progress at version 20304 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.94s no progress at version 1145260 (avg 0.86s) [limit 16].
Test Ok

github-actions · 2025-02-10T18:38:39Z

✅ Forge suite `compat` success on `255673e8f2757ed8634a7d65d7aca11de7016cde` ==> `304ce07afe7115510b88ca8c136d8ae909d12d14`

Compatibility test results for 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 304ce07afe7115510b88ca8c136d8ae909d12d14 (PR)
1. Check liveness of validators at old version: 255673e8f2757ed8634a7d65d7aca11de7016cde
compatibility::simple-validator-upgrade::liveness-check : committed: 4169.97 txn/s, submitted: 4170.13 txn/s, expired: 0.16 txn/s, latency: 2949.35 ms, (p50: 2700 ms, p70: 3000, p90: 4200 ms, p99: 6700 ms), latency samples: 363166
2. Upgrading first Validator to new version: 304ce07afe7115510b88ca8c136d8ae909d12d14
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 3248.89 txn/s, latency: 9042.57 ms, (p50: 9600 ms, p70: 11000, p90: 11900 ms, p99: 12000 ms), latency samples: 70620
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 3285.70 txn/s, latency: 10485.38 ms, (p50: 11500 ms, p70: 11800, p90: 11900 ms, p99: 12100 ms), latency samples: 126920
3. Upgrading rest of first batch to new version: 304ce07afe7115510b88ca8c136d8ae909d12d14
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 3342.65 txn/s, latency: 8896.32 ms, (p50: 9500 ms, p70: 10700, p90: 11400 ms, p99: 11500 ms), latency samples: 74300
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3312.36 txn/s, latency: 10431.14 ms, (p50: 11300 ms, p70: 11500, p90: 11600 ms, p99: 11800 ms), latency samples: 127160
4. upgrading second batch to new version: 304ce07afe7115510b88ca8c136d8ae909d12d14
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 5106.76 txn/s, latency: 5895.58 ms, (p50: 6500 ms, p70: 6700, p90: 7800 ms, p99: 7900 ms), latency samples: 101320
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 5088.60 txn/s, latency: 6599.39 ms, (p50: 6800 ms, p70: 7700, p90: 8300 ms, p99: 8400 ms), latency samples: 181080
5. check swarm health
Compatibility test for 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 304ce07afe7115510b88ca8c136d8ae909d12d14 passed
Test Ok

georgemitenkov requested review from vineethk and igor-aptos January 29, 2025 14:29

georgemitenkov marked this pull request as ready for review January 29, 2025 14:35

georgemitenkov force-pushed the george/replay-benchmark-package-override branch from 731260e to c65a196 Compare January 29, 2025 14:36

vineethk approved these changes Jan 29, 2025

View reviewed changes

georgemitenkov requested a review from gelash January 29, 2025 16:17

georgemitenkov force-pushed the george/replay-benchmark-package-override branch 2 times, most recently from 103467a to a8fc584 Compare January 29, 2025 16:52

georgemitenkov requested review from msmouse and wrwg January 30, 2025 10:56

georgemitenkov force-pushed the george/replay-benchmark-package-override branch from a8fc584 to 5e3e076 Compare February 6, 2025 09:31

This was referenced Feb 6, 2025

[benchmark] Make paranoid mode configurable #15900

Merged

[benchmark] Make gas metering conditional #15901

Open

gelash approved these changes Feb 6, 2025

View reviewed changes

georgemitenkov added 4 commits February 8, 2025 16:39

[benchmark] Add option for package overrides

657bca1

update readme

c3e00d0

comments: address simple comments

e3b31f4

address comments

c8171a9

georgemitenkov force-pushed the george/replay-benchmark-package-override branch from 5e3e076 to c8171a9 Compare February 8, 2025 16:55

georgemitenkov enabled auto-merge (squash) February 8, 2025 16:56

This comment has been minimized.

Sign in to view

fix lint

304ce07

This comment has been minimized.

Sign in to view

georgemitenkov merged commit b64e048 into main Feb 10, 2025
43 of 46 checks passed

georgemitenkov deleted the george/replay-benchmark-package-override branch February 10, 2025 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[benchmark] Add option for package overrides #15841

[benchmark] Add option for package overrides #15841

georgemitenkov commented Jan 29, 2025 •

edited

Loading

trunk-io bot commented Jan 29, 2025 •

edited

Loading

georgemitenkov commented Jan 29, 2025 •

edited

Loading

vineethk left a comment

vineethk Jan 29, 2025

georgemitenkov Jan 29, 2025

vineethk Jan 29, 2025

georgemitenkov Jan 29, 2025

gelash Feb 6, 2025

georgemitenkov Feb 8, 2025

gelash Feb 6, 2025

georgemitenkov Feb 8, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 10, 2025

This comment has been minimized.

github-actions bot commented Feb 10, 2025

[benchmark] Add option for package overrides #15841

[benchmark] Add option for package overrides #15841

Conversation

georgemitenkov commented Jan 29, 2025 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Jan 29, 2025 • edited Loading

georgemitenkov commented Jan 29, 2025 • edited Loading

vineethk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Feb 10, 2025

✅ Forge suite realistic_env_max_load success on 304ce07afe7115510b88ca8c136d8ae909d12d14

This comment has been minimized.

github-actions bot commented Feb 10, 2025

✅ Forge suite compat success on 255673e8f2757ed8634a7d65d7aca11de7016cde ==> 304ce07afe7115510b88ca8c136d8ae909d12d14

georgemitenkov commented Jan 29, 2025 •

edited

Loading

trunk-io bot commented Jan 29, 2025 •

edited

Loading

georgemitenkov commented Jan 29, 2025 •

edited

Loading

✅ Forge suite `realistic_env_max_load` success on `304ce07afe7115510b88ca8c136d8ae909d12d14`

✅ Forge suite `compat` success on `255673e8f2757ed8634a7d65d7aca11de7016cde` ==> `304ce07afe7115510b88ca8c136d8ae909d12d14`