`Development`: Adjust test coverage thresholds #10485

MaximilianAnzinger · 2025-03-15T22:17:30Z

Checklist

General

I chose a title conforming to the naming conventions for pull requests.

Motivation & Description

Our test coverage checks on develop are currently failing.
This PR adjusts the coverage thresholds so our CI pipelines pass again.

Code Review

Code Review 1
Code Review 2

Summary by CodeRabbit

Chores
- Fine-tuned internal quality metrics by adjusting test coverage thresholds, ensuring our development practices remain robust and up-to-date.
- These updates improve our quality assurance process without changing the external behavior or user experience of the application.

raffifasaro

code LGTM

coderabbitai · 2025-03-15T22:19:42Z

Walkthrough

This pull request modifies the gradle/jacoco.gradle file by adjusting coverage thresholds for various modules. The changes include decreasing the aggregated instruction threshold and specific module thresholds for "athena", "buildagent", "core", "plagiarism", and "programming". The class coverage thresholds for some modules have also been increased. No changes were made to exported or public entities.

Changes

File	Change Summary
gradle/jacoco.gradle	Adjusted coverage thresholds: AggregatedCoverageThresholds for "INSTRUCTION" decreased from 0.890 to 0.880; "CLASS" increased from 69 to 70. "athena" from 0.891 to 0.874; "buildagent" from 0.846 to 0.841; "core" "INSTRUCTION" from 0.861 to 0.850 and "CLASS" from 16 to 17; "plagiarism" from 0.910 to 0.900; "programming" from 0.872 to 0.870.

Possibly related PRs

Development: Reduce coverage ratios to fix failing checks #10239: The changes in the main PR and the retrieved PR both involve modifications to the gradle/jacoco.gradle file, specifically adjusting coverage thresholds, indicating a direct relationship at the code level.
Development: Improve JaCoCo coverage report and validation tasks #10299: The changes in the main PR regarding adjustments to coverage thresholds in gradle/jacoco.gradle are directly related to the modifications in the retrieved PR, which also involves changes to coverage thresholds in the same file.

Suggested labels

ready for review, chore

Suggested reviewers

krusche
SimonEntholzer
HanyangXu0508
tobias-lippert

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)

We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
- To enable this feature, set early_access to true under in the settings.

📜 Recent review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 45c5cdf and f8daf86.

📒 Files selected for processing (1)

gradle/jacoco.gradle (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

gradle/jacoco.gradle

⏰ Context from checks skipped due to timeout of 90000ms (7)

GitHub Check: Call Build Workflow / Build .war artifact
GitHub Check: Call Build Workflow / Build and Push Docker Image
GitHub Check: server-style
GitHub Check: client-style
GitHub Check: client-tests
GitHub Check: server-tests
GitHub Check: Analyse

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai or @coderabbitai title anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

gradle/jacoco.gradle (1)

3-3: Revisit AggregatedCoverageThresholds Configuration:
The TODO comment suggests that AggregatedCoverageThresholds should be set to 90% for INSTRUCTION and 0 for CLASS, but the current values are 0.880 for INSTRUCTION and 69 for CLASS. Please verify whether the CLASS threshold value is intentionally maintained at 69 or if it needs to be updated to 0 to meet the intended requirements.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c6730a5 and eb62b72.

📒 Files selected for processing (1)

gradle/jacoco.gradle (2 hunks)

⏰ Context from checks skipped due to timeout of 90000ms (7)

GitHub Check: Call Build Workflow / Build and Push Docker Image
GitHub Check: Call Build Workflow / Build .war artifact
GitHub Check: client-tests
GitHub Check: server-tests
GitHub Check: server-style
GitHub Check: client-style
GitHub Check: Analyse

🔇 Additional comments (4)

gradle/jacoco.gradle (4)

7-7: Verify Athena Module Thresholds:
The athena module thresholds have been updated to 0.874 (INSTRUCTION) and 2 (CLASS). Please confirm that these adjusted thresholds are aligned with the project's updated coverage requirements.

9-9: Confirm Buildagent Module Thresholds Adjustment:
The buildagent module now has an INSTRUCTION threshold of 0.841 and CLASS set to 0. Ensure that this change is intentional and consistent with the overall coverage policy.

19-19: Plagiarism Module Threshold Update:
The INSTRUCTION threshold for the plagiarism module has been lowered to 0.900 while CLASS remains at 0. This adjustment appears consistent with a reduction in the required coverage. Please verify that these values meet the project's quality expectations.

20-20: Programming Module Threshold Tweak:
There is a minor adjustment in the programming module’s INSTRUCTION threshold—from what is presumed to be 0.872 to 0.870. Please confirm that this subtle change satisfies the updated coverage guidelines.

tobias-lippert

appropriate fix for now

jfr2102 · 2025-03-17T16:46:08Z

Coverage dropped further since then: https://bamboo.ase.in.tum.de/browse/ARTEMIS-TESTS-JAVATEST-6959:

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':jacocoTestCoverageVerification-aggregated'.
> Rule violated for bundle Artemis: instructions covered ratio is 0.86, but expected minimum is 0.89
  Rule violated for bundle Artemis: classes missed count is 86, but expected maximum is 69

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':jacocoTestCoverageVerification-athena'.
> Rule violated for bundle Artemis: instructions covered ratio is 0.874, but expected minimum is 0.891

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':jacocoTestCoverageVerification-communication'.

jfr2102

to pass coverage on develop we would need to go lower for some: https://bamboo.ase.in.tum.de/browse/ARTEMIS-TESTS-JAVATEST-6959

jfr2102 · 2025-03-17T16:50:25Z

gradle/jacoco.gradle

@@ -1,23 +1,23 @@
 ext {
    // TODO: this should become 90% for INSTRUCTION and 0 for CLASS
-    AggregatedCoverageThresholds = ["INSTRUCTION": 0.890, "CLASS": 69]
+    AggregatedCoverageThresholds = ["INSTRUCTION": 0.880, "CLASS": 70]


Suggested change

AggregatedCoverageThresholds = ["INSTRUCTION": 0.880, "CLASS": 70]

AggregatedCoverageThresholds = ["INSTRUCTION": 0.86, "CLASS": 86]

jfr2102 · 2025-03-17T16:53:10Z

gradle/jacoco.gradle

        "atlas"         :          ["INSTRUCTION": 0.905, "CLASS":  5],
-        "buildagent"    :          ["INSTRUCTION": 0.846, "CLASS":  0],
+        "buildagent"    :          ["INSTRUCTION": 0.841, "CLASS":  0],
        "communication" :          ["INSTRUCTION": 0.916, "CLASS":  2],


Suggested change

"communication" : ["INSTRUCTION": 0.916, "CLASS": 2],

"communication" : ["INSTRUCTION": 0.915, "CLASS": 2],

MaximilianAnzinger · 2025-03-18T09:47:16Z

I'll close this since we're looking for a sustainable solution to automate this process with a script.

adjust coverage thresholds

eb62b72

MaximilianAnzinger requested a review from a team as a code owner March 15, 2025 22:17

github-actions bot assigned MaximilianAnzinger Mar 15, 2025

raffifasaro previously approved these changes Mar 15, 2025

View reviewed changes

coderabbitai bot reviewed Mar 15, 2025

View reviewed changes

coderabbitai bot previously approved these changes Mar 15, 2025

View reviewed changes

tobias-lippert previously approved these changes Mar 16, 2025

View reviewed changes

adjust core coverage

45c5cdf

MaximilianAnzinger dismissed stale reviews from tobias-lippert, coderabbitai[bot], and raffifasaro via 45c5cdf March 16, 2025 11:15

coderabbitai bot previously approved these changes Mar 16, 2025

View reviewed changes

adjust assessment

f8daf86

MaximilianAnzinger dismissed coderabbitai[bot]’s stale review via f8daf86 March 16, 2025 12:03

coderabbitai bot approved these changes Mar 16, 2025

View reviewed changes

tobias-lippert approved these changes Mar 16, 2025

View reviewed changes

Hialus approved these changes Mar 16, 2025

View reviewed changes

jfr2102 requested changes Mar 17, 2025

View reviewed changes

MaximilianAnzinger closed this Mar 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`Development`: Adjust test coverage thresholds #10485

`Development`: Adjust test coverage thresholds #10485

MaximilianAnzinger commented Mar 15, 2025 •

edited by coderabbitai bot

Loading

raffifasaro left a comment

coderabbitai bot commented Mar 15, 2025 •

edited

Loading

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

coderabbitai bot left a comment

tobias-lippert left a comment

jfr2102 commented Mar 17, 2025

jfr2102 left a comment

jfr2102 Mar 17, 2025

jfr2102 Mar 17, 2025

MaximilianAnzinger commented Mar 18, 2025

	AggregatedCoverageThresholds = ["INSTRUCTION": 0.880, "CLASS": 70]
	AggregatedCoverageThresholds = ["INSTRUCTION": 0.86, "CLASS": 86]

	"communication" : ["INSTRUCTION": 0.916, "CLASS": 2],
	"communication" : ["INSTRUCTION": 0.915, "CLASS": 2],

Development: Adjust test coverage thresholds #10485

Development: Adjust test coverage thresholds #10485

Conversation

MaximilianAnzinger commented Mar 15, 2025 • edited by coderabbitai bot Loading

Checklist

General

Motivation & Description

Code Review

Summary by CodeRabbit

raffifasaro left a comment

Choose a reason for hiding this comment

coderabbitai bot commented Mar 15, 2025 • edited Loading

Walkthrough

Changes

Possibly related PRs

Suggested labels

Suggested reviewers

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

Documentation and Community

coderabbitai bot left a comment

Choose a reason for hiding this comment

tobias-lippert left a comment

Choose a reason for hiding this comment

jfr2102 commented Mar 17, 2025

jfr2102 left a comment

Choose a reason for hiding this comment

jfr2102 Mar 17, 2025

Choose a reason for hiding this comment

jfr2102 Mar 17, 2025

Choose a reason for hiding this comment

MaximilianAnzinger commented Mar 18, 2025

`Development`: Adjust test coverage thresholds #10485

`Development`: Adjust test coverage thresholds #10485

MaximilianAnzinger commented Mar 15, 2025 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 15, 2025 •

edited

Loading