Add execution time metrics

[HansRobo]

Description

Abstract

This pull-request adds computation time metrics for...

• allowing scenarios to evaluate the impact of increased computation time
• allowing us to find out the calculation time and analyze with rosbags

Additionally, this pull-request added test scenario that has execution time conditions.

Background

Previously, scenario_simulator_v2 simply dispaly a log like below if the main loop is timed out:

Your machine is not powerful enough to run the scenario at the specified frame rate

This pull-reuqest has been made to allow for more detailed analysis.

Details

added metrics topics

• /simulation/interpreter/execution_time/evaluate
  • execution time for evaluating scenario
• /simulation/interpreter/execution_time/update
  • execution time for updating traffic_simulator
• /simulation/interpreter/execution_time/output
  • execution time for creating and publishing simulation context

optional scenarios in workflow

This pull-request added execution_time_test.yaml, but the execution time and the scenario result are flaky.
Off cource, greater execution time threshold make the scenarios stable.
But the greater the threshold, the lesshoh value there is in the scenario.

In this pull-request, I prepare optional scenario workflows for effective but flaky scenarios.
The scenario failures don't affect CI result, but report the result and error messages by comment as below.

References

Regression Test: OK

Destructive Changes

None

Known Limitations

None

[github-actions]

Checklist for reviewers ☑️

All references to "You" in the following text refer to the code reviewer.

• Is this pull request written in a way that is easy to read from a third-party perspective?
• Is there sufficient information (background, purpose, specification, algorithm description, list of disruptive changes, and migration guide) in the description of this pull request?
• If this pull request contains a destructive change, does this pull request contain the migration guide?
• Labels of this pull request are valid?
• All unit tests/integration tests are included in this pull request? If you think adding test cases is unnecessary, please describe why and cross out this line.
• The documentation for this pull request is enough? If you think adding documents for this pull request is unnecessary, please describe why and cross out this line.

[github-actions]

Failure optional scenarios

This is an experimental check and does not block merging the pull-request. But, please be aware that this may indicate a regression.

scenario failed: execution_time_test.junit.xml

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002308999999999999854644050501) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[github-actions]

Failure optional scenarios

This is an experimental check and does not block merging the pull-request. But, please be aware that this may indicate a regression.

scenario failed: execution_time_test.junit.xml

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002164999999999999823890872719) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[github-actions]

Failure optional scenarios

Note

This is an experimental check and does not block merging the pull-request.
But, please be aware that this may indicate a regression.

scenario failed: execution_time_test.junit.xml

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002955999999999999860306187927) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[github-actions]

Failure optional scenarios

Note

This is an experimental check and does not block merging the pull-request.
But, please be aware that this may indicate a regression.

scenario failed: execution_time_test.junit.xml

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002618000000000000101335606573) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[github-actions]

Failure optional scenarios

Note

This is an experimental check and does not block merging the pull-request.
But, please be aware that this may indicate a regression.

scenario failed: execution_time_test

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002360000000000000118516307879) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[github-actions]

Failure optional scenarios

Note

This is an experimental check and does not block merging the pull-request.
But, please be aware that this may indicate a regression.

scenario failed: execution_time_test

      <failure type="SimulationFailure" message="CustomCommandAction typed &quot;exitFailure&quot; was triggered by the named Conditions {&quot;update time checker&quot;, &quot;avoid startup&quot;}: {&quot;update time checker&quot;: Is the /simulation/interpreter/execution_time/update (= 0.002327000000000000075328632221) is greaterThan 0.0?}, {&quot;avoid startup&quot;: Is the simulation time (= 1.000000000000000222044604925031) is greaterThan 1.000000000000000000000000000000?}" />

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud