Upload code coverage #7262

stephanos · 2025-02-06T00:11:06Z

What changed?

Fixed code coverage reporting.

Why?

We haven't reported code coverage in over a year.

It stopped working when we moved away from Buildkite.

How did you test it?

Checkout https://app.codecov.io/gh/temporalio/temporal

There's no data for main ("master") yet, though, until we merge.

I'll also leave the README coveralls badge there for now.

Potential risks

Documentation

Is hotfix candidate?

stephanos · 2025-03-12T19:05:14Z

Makefile

-.PHONY: $(SUMMARY_COVER_PROFILE)
-$(SUMMARY_COVER_PROFILE):
-	@printf $(COLOR) "Combine coverage reports to $(SUMMARY_COVER_PROFILE)..."
-	@rm -f $(SUMMARY_COVER_PROFILE) $(SUMMARY_COVER_PROFILE).html
-	@if [ -z "$(wildcard $(TEST_OUTPUT_ROOT)/*.cover.out)" ]; then \
-		echo "No coverage data, aborting!" && exit 1; \
-	fi
-	@echo "mode: atomic" > $(SUMMARY_COVER_PROFILE)
-	$(foreach COVER_PROFILE,$(wildcard $(TEST_OUTPUT_ROOT)/*.cover.out),\
-		@printf "Add %s...\n" $(COVER_PROFILE); \
-		@grep -v -e "[Mm]ocks\?.go" -e "^mode: \w\+" $(COVER_PROFILE) >> $(SUMMARY_COVER_PROFILE) || true \
-	$(NEWLINE))
-
-coverage-report: $(SUMMARY_COVER_PROFILE)
-	@printf $(COLOR) "Generate HTML report from $(SUMMARY_COVER_PROFILE) to $(SUMMARY_COVER_PROFILE).html..."
-	@go tool cover -html=$(SUMMARY_COVER_PROFILE) -o $(SUMMARY_COVER_PROFILE).html


None of this is needed/used. Codecov aggregates all reports for us. And its config allows to filter out irrelevant files/folders.

stephanos · 2025-03-12T19:05:59Z

.github/workflows/run-tests.yml

@@ -60,7 +60,6 @@ env:
  PR_BASE_COMMIT: ${{ github.event.pull_request.base.sha }}
  DOCKER_COMPOSE_FILE: ./develop/github/docker-compose.yml
  TEMPORAL_VERSION_CHECK_DISABLED: 1
-  BUILDKITE_ANALYTICS_TOKEN: ${{ secrets.BUILDKITE_ANALYTICS_TOKEN }}


Can I delete then?

stephanos · 2025-03-12T19:08:24Z

Makefile

 NEW_REPORT              = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml   # generates a new filename each time it's substituted
+COVERPKG_FLAG 		    = -coverpkg=$(shell go list ./... | paste -sd "," -)


Just listing all packages here. Anything we don't want to be reported is filtered out by Codecov.

The only downside is that it'll generate a bunch of warning: no packages being tested depend on matches for pattern - but I think that's a small price to pay for not having to maintain this list by hand (and risking it gets out of date).

I tried to keep number of Linux commands under control but it seems to get out of it long time ago (this is comment about adding dependency in paste).

Is the alternative to write a Go script? I feel like there's already way too many Unix-specific stuff in this Makefile already. (e.g. /dev/urandom just above it)

stephanos · 2025-03-12T20:56:12Z

Makefile

-NEW_COVER_PROFILE       = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).cover.out   # generates a new filename each time it's substituted
-SUMMARY_COVER_PROFILE  := $(TEST_OUTPUT_ROOT)/summary.cover.out
-NEW_REPORT              = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml   # generates a new filename each time it's substituted
+NEW_COVER_PROFILE       = $(TEST_OUTPUT_ROOT)/coverage.$(shell xxd -p -l 16 /dev/urandom).out   # generates a new filename each time it's substituted


File name has to match any item in this list to be found by Codecov.

stephanos · 2025-03-12T20:56:35Z

Makefile

-SUMMARY_COVER_PROFILE  := $(TEST_OUTPUT_ROOT)/summary.cover.out
-NEW_REPORT              = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml   # generates a new filename each time it's substituted
+NEW_COVER_PROFILE       = $(TEST_OUTPUT_ROOT)/coverage.$(shell xxd -p -l 16 /dev/urandom).out   # generates a new filename each time it's substituted
+NEW_REPORT              = $(TEST_OUTPUT_ROOT)/junit.$(shell xxd -p -l 16 /dev/urandom).xml   # generates a new filename each time it's substituted


Aligning the file name pattern here with the coverage one.

I don't expect that to break anything?

We will see :-)

alexshtin · 2025-03-13T02:01:34Z

.github/workflows/run-tests.yml

@@ -60,7 +60,6 @@ env:
  PR_BASE_COMMIT: ${{ github.event.pull_request.base.sha }}
  DOCKER_COMPOSE_FILE: ./develop/github/docker-compose.yml
  TEMPORAL_VERSION_CHECK_DISABLED: 1
-  BUILDKITE_ANALYTICS_TOKEN: ${{ secrets.BUILDKITE_ANALYTICS_TOKEN }}


Can I delete then?

alexshtin · 2025-03-13T02:03:25Z

Makefile

-SUMMARY_COVER_PROFILE  := $(TEST_OUTPUT_ROOT)/summary.cover.out
-NEW_REPORT              = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml   # generates a new filename each time it's substituted
+NEW_COVER_PROFILE       = $(TEST_OUTPUT_ROOT)/coverage.$(shell xxd -p -l 16 /dev/urandom).out   # generates a new filename each time it's substituted
+NEW_REPORT              = $(TEST_OUTPUT_ROOT)/junit.$(shell xxd -p -l 16 /dev/urandom).xml   # generates a new filename each time it's substituted


We will see :-)

alexshtin · 2025-03-13T02:05:20Z

Makefile

 NEW_REPORT              = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml   # generates a new filename each time it's substituted
+COVERPKG_FLAG 		    = -coverpkg=$(shell go list ./... | paste -sd "," -)


I tried to keep number of Linux commands under control but it seems to get out of it long time ago (this is comment about adding dependency in paste).

alexshtin · 2025-03-13T16:51:49Z

tools/testrunner/testrunner.go

+const (
+	codeCoverageExtension = ".cover.out"
+	retriesFlag           = "--retries="
+	coverProfileFlag      = "-coverprofile="


why this one has only one dash?

coverprofile is from Go, --retriesFlag and --junitfile are out own invention. Not sure which one is "correct".

stephanos force-pushed the fix-coveralls branch 30 times, most recently from de3cf47 to a7721fc Compare February 7, 2025 00:54

stephanos force-pushed the fix-coveralls branch from ddf442c to 0fa86c2 Compare March 12, 2025 19:04

stephanos commented Mar 12, 2025

View reviewed changes

stephanos force-pushed the fix-coveralls branch from 0fa86c2 to c0ff644 Compare March 12, 2025 19:08

stephanos commented Mar 12, 2025

View reviewed changes

stephanos force-pushed the fix-coveralls branch 5 times, most recently from c838d73 to 5a32618 Compare March 12, 2025 20:54

stephanos commented Mar 12, 2025

View reviewed changes

stephanos force-pushed the fix-coveralls branch from 5a32618 to 03e8f13 Compare March 12, 2025 21:17

temporalio deleted a comment from codecov bot Mar 12, 2025

stephanos force-pushed the fix-coveralls branch 4 times, most recently from 0045835 to 01d652f Compare March 12, 2025 21:45

stephanos marked this pull request as ready for review March 12, 2025 21:46

stephanos requested a review from a team as a code owner March 12, 2025 21:46

stephanos requested review from alexshtin and ychebotarev March 12, 2025 21:46

Upload code coverage

a70048d

stephanos force-pushed the fix-coveralls branch from 01d652f to a70048d Compare March 12, 2025 21:54

drop leading slash

2b889ea

ychebotarev approved these changes Mar 12, 2025

View reviewed changes

stephanos added 2 commits March 13, 2025 08:38

tweak ignore list

07c3cc9

link to ignore list docs

a24a9e4

alexshtin approved these changes Mar 13, 2025

View reviewed changes

stephanos merged commit b609189 into temporalio:main Mar 13, 2025
51 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upload code coverage #7262

Upload code coverage #7262

stephanos commented Feb 6, 2025 •

edited

Loading

stephanos Mar 12, 2025 •

edited

Loading

stephanos Mar 12, 2025

alexshtin Mar 13, 2025

stephanos Mar 13, 2025

stephanos Mar 12, 2025 •

edited

Loading

alexshtin Mar 13, 2025

stephanos Mar 13, 2025

stephanos Mar 12, 2025

stephanos Mar 12, 2025 •

edited

Loading

alexshtin Mar 13, 2025

alexshtin Mar 13, 2025

alexshtin Mar 13, 2025

alexshtin Mar 13, 2025

alexshtin Mar 13, 2025

stephanos Mar 13, 2025

		NEW_REPORT = $(TEST_OUTPUT_ROOT)/$(shell xxd -p -l 16 /dev/urandom).junit.xml # generates a new filename each time it's substituted
		COVERPKG_FLAG = -coverpkg=$(shell go list ./... \| paste -sd "," -)

Upload code coverage #7262

Upload code coverage #7262

Conversation

stephanos commented Feb 6, 2025 • edited Loading

What changed?

Why?

How did you test it?

Potential risks

Documentation

Is hotfix candidate?

stephanos Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanos Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanos Mar 12, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanos commented Feb 6, 2025 •

edited

Loading

stephanos Mar 12, 2025 •

edited

Loading

stephanos Mar 12, 2025 •

edited

Loading

stephanos Mar 12, 2025 •

edited

Loading