TEST_RESULTS

How does a person read the stability and the timing of the tests of an app?

Binds: code

xcloud.stability and xcloud.timing read the result bundles that the caller saves after each Deliver build. Each bundle answers one row for each test, destination and build. tools/apple-app-delivery/docs/README.md, section “Each application keeps its own cache of result bundles”, states where the rows come from.

A report reads the newest build alone. xcloud.prune deletes the artifacts of each build older than the newest one. So a report reads no older build.

Stability puts a warning rate beside the failure rate

xcloud.stability ranks the rows by failure rate. A test can also carry a warning and still pass. xcresulttool reports such a warning as a Runtime Warning child of the node it happened on. It changes nothing about the Passed result of that node. So the ranking puts a warning rate beside the failure rate. A test that always recovers on a retry then stays visible.

The two rates count different attempts. A failure rate reads every attempt, because every build can fail. A warning rate reads the attempts of a build that carries the code that records the warning. first_warning_build in Settings names the first such build. The wruns column states how many attempts the warning rate rests on.

xcloud.stability reads no month, because a failure rate carries no month.

Timing compares a month with the month before it

xcloud.timing reports a median and a slowest run for each test on each destination. It also reports one number for a whole run. That number is the sum of the test durations of one build. The report takes the median of that sum across the builds of a month, per destination. The median stops a month with more builds from reading as a slower month. The comparison of a month with the month before it says whether the suite grows.

An artifact carries the month. A build number carries none. GitHub answers created_at on each artifact. xcloud.timing groups by the calendar month of that field.

A duration in the xcloud.timing report covers the test, and not the time before it. xcresulttool answers durationInSeconds on each test. That field counts the app launch. It counts the time after Tear Down starts too. It leaves out the time before the test starts. So the total reads lower than a stopwatch on the same suite, and the two never match.

Every number in the xcloud.timing report comes from Xcode Cloud. A run on a Mac at a desk saves no artifact, and the report reads nothing of it. So the report says nothing about a laptop.