WordPress.org

Make WordPress Core

Opened 7 months ago

Closed 7 months ago

#52561 closed defect (bug) (invalid)

Only run and upload one test run to Codecov

Reported by: johnbillion Owned by:
Milestone: Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: needs-patch
Focuses: Cc:

Description

In the Code Coverage Report workflow, the test suite is run once for single site and once for Multisite and both results are uploaded to Codecov. It appears that Codecov only provides graphs for one upload from each commit, and it appears that the most recent is used, which means only the Multisite report is used as it runs second. Example: https://codecov.io/gh/WordPress/wordpress-develop/tree/f400eb5ab9e80f62939bd8630e27e08ee0619ba1/src

This means the single site test run in this workflow is useless.

Let's remove either the single site test run or the Multisite test run to avoid CPU and speed up this workflow.

My opinion is we should stick with the Multisite test run so we have coverage reports for Multisite files too.

Change History (4)

#1 @desrosj
7 months ago

This makes sense, and using the Multisite report sounds good.

#2 @swissspidy
7 months ago

Would Codecov‘s flag feature help here?

I remember discussing this before in another ticket.

https://docs.codecov.io/docs/flags

#3 @jorbin
7 months ago

We are using the flags and they are working as expected though the UI in codecov could lead to some confusion here. I'll use wp_lostpassword_url as an example for why I think things are working as expected.

By default, all three flags are checked and we can see that line 624 is covered:

https://ucacf470ce8dc58cbe0ad122e7b8.previews.dropboxusercontent.com/p/thumb/ABEJ0dX3RexN1FGgWjfSeO0XZVWnTMJIdl8Da3pPhaljG2iY3ejk5abSfEC_K235Fe64DN8fis0VGLHbtGVA99PSGioi04zpFUQrDLf7Jxk5S1cyFIxs_fWeCDrEe4625NlovRqHiiJcI5vCnVhmggf6GShpU9vOO7FqIyTnFhP64a16nc6pST8VXFPDBJQftMvXiYrJ3tIpcVONuY_amTVYBGopAHcVQilpC6p9qcMH_1Pq3OCK4oBwzmlSV-zBAJwOzVDCyfDsTi3xlLm15nmE7KgyzOgqJcS1Ud8PEMVTp2E2CBMCNiHwUkHGuxotF3UUUcUgHmhVOzvqvTEBPWERZwteSMTnn4U-qgIdhWvi1KE_S-YY_LclSQwXm7_3g8_1euiJue0u38f7ezN9e2Mf/p.png

If only single is selected, then 624 is not covered, but 627 remains covered:
https://uc5f1bd0cf57db3184437bf7e979.previews.dropboxusercontent.com/p/thumb/ABE9o25zFnCPGDj84zucMeI6vtiI1cxCgk5inaY8JjH4WnqQ48pe9ov8A45KsLBB6VlXgGD8uSTA-idvfRXOR7Y9QezhkZuRn1shbtOcYV9VxVzzbdS-fBBV5bw09E4UnXi7e0FrVUnDTvoQClCzzGwat8ZscB__KxIcFuAuqHkrLb8uA5z4tohAHuTfNIjJl3y71QDvdD2HwBw3ETyPc7_zEoIOUjcvmxa8_nn8IAOxd0ZWKzuoz5GmGYWWEE24uPt0Mdzst6_kNOFsMNlR6dsv5Gf7gTI4hKShYeETufLgH3b5CdTjTPTeZ_VtzzMkFWuG1HdtwJvABcIUoGSPldYzKeVOu6dF5CQW1Z79AmNuEPiekrmzVdseO3Rc6f9oKINV49BRMRrCzkJJ4xaCWYrt/p.png

If only multisite is selected, then 624 is still covered, but 627 is not.

https://uc171d0fa1edc272379c53ce488e.previews.dropboxusercontent.com/p/thumb/ABHWgNG74VXAUaRadhGYEbQVvUAqiFOZFZqhJPPR7NaU8_I6v8D46G3Kw-s4Qq0dnaB65y-Omq7kkeIE6EXwijrJdjfGbL2yOIw7MOV7OVRhGBe6dLg6DrIy9dNKUn2KhOOouVwFdonR3nyN4yPj3cp32L1kt49Y2COjHBv-sB2pbntynmPeEMaQbpV6wxxP_q8i68ZfYkOnSn840cV9k9wkfUjyeqgY1G5IcVX5O7SvL5tn-j-ZFF8Bxa07rss_z58n-HPzZu-raINWg1_Uobk7EAYz-I22xqKrtU1IOhx3fZQt4EL-YqB6rCqEqr7tublZckWhfPxkrCQyjJN9Z6atT2TbdQcZpp2F-rDwG0cgUONG32DJmSNJyn7yueRHaZm_r7qPGGNDUHaF_EH6BYOj/p.png

The reason there are three flags is so that when we add E2E tests, we can see what is covered by php tests and what is covered by e2e tests. I don't know why there is a 2 in the sidebar when 3 flags should have a line covered.

#4 @johnbillion
7 months ago

  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

Thanks for that info @jorbin! I hadn't seen the flags on the single file views because I was looking at the Coverage numbers on the directory listing view where unfortunately those filters aren't available.

This does indeed seem to be working as expected even if the UI is not completely clear.

I'll close this off.

Note: See TracTickets for help on using tickets.