CI: run what a change needs, and the browser profiles in parallel

littlefox commented

2026-10-27 18:06:11 +00:00

Owner

The user (D-070): "CI runs take very long. For things that do not necessarily need to take that long (for example merging documentation and roadmap). Investigate ways to speed up the CI workflow by introducing sound logic for what tests to run when."

Where the time goes (run #619, typical)

A ci run takes 36–40 minutes, and every PR needs two (branch, then main): ~75 minutes per merge.

Job	Time
check (clippy ×2, deny, unit, integration)	~6 min
desktop (clippy, unit, smoke)	~7 min
e2e	~35 min: web build 4.5, then 429 tests on one worker: desktop 10.1, android 8.2, iphone 10.7 min, one after another

The three profiles already have separate servers, relays and page hosts (#64); only workers: 1 keeps them in sequence.

Last 60 merges to main: 7 docs-only, 2 tests/infra-only, 51 code. 41 of the 60 main runs tested a tree identical to the branch head that had already passed (the branch was up to date with main).

Proposal

Profiles in parallel. A web-build job builds the release web app and relay once and uploads them; three e2e jobs (desktop, android, iphone), each with its own lyskomd service, download them and run their project. e2e goes from ~35 to ~15 min wall time, for every run that needs it.
A changes job first (tools/ci-plan.py, unit-tested) decides which jobs run. The rules fail safe: any file not covered by a narrower rule runs everything.
- Only docs (docs/** except docs/protocol/**, *.md, LICENSE): nothing heavy.
- Only e2e/** / tools/e2e.sh: the e2e jobs.
- Desktop job only when app/, crates/, Cargo.*, devserver/, ci/, the smoke script or the workflow changed.
- A branch is compared with its merge base on main (the whole branch, not the last commit).
- On main: a merge commit whose tree equals its second parent's (the PR head) was tested as-is, and pr-merge only merges when all its runs passed; confirmed through the API, else everything runs. Otherwise the merge's whole diff is classified as above.
- workflow_dispatch: everything. New: a nightly full run on main, so nothing skipped stays untested for more than a day.
Later, measured first: cache ~/.cargo/registry and build output on the shared runners (each job downloads every crate today).

Expected: a code PR ~17 min on the branch and ~1 min on main (most of the time); a docs/roadmap PR ~1–2 min each.

CLAUDE.md's workflow section is updated to match (it also says main's run includes the soak test; that's the nightly soak workflow, while the e2e run has the short soak spec).

The user (D-070): "CI runs take very long. For things that do not necessarily need to take that long (for example merging documentation and roadmap). Investigate ways to speed up the CI workflow by introducing sound logic for what tests to run when." ## Where the time goes (run #619, typical) A `ci` run takes 36–40 minutes, and every PR needs two (branch, then main): ~75 minutes per merge. | Job | Time | |---|---| | check (clippy ×2, deny, unit, integration) | ~6 min | | desktop (clippy, unit, smoke) | ~7 min | | **e2e** | **~35 min**: web build 4.5, then 429 tests on **one worker**: desktop 10.1, android 8.2, iphone 10.7 min, one after another | The three profiles already have separate servers, relays and page hosts (#64); only `workers: 1` keeps them in sequence. Last 60 merges to main: 7 docs-only, 2 tests/infra-only, 51 code. **41 of the 60 main runs tested a tree identical to the branch head that had already passed** (the branch was up to date with main). ## Proposal 1. **Profiles in parallel.** A `web-build` job builds the release web app and relay once and uploads them; three e2e jobs (desktop, android, iphone), each with its own lyskomd service, download them and run their project. e2e goes from ~35 to ~15 min wall time, for every run that needs it. 2. **A `changes` job first** (`tools/ci-plan.py`, unit-tested) decides which jobs run. The rules fail safe: any file not covered by a narrower rule runs everything. - Only docs (`docs/**` except `docs/protocol/**`, `*.md`, `LICENSE`): nothing heavy. - Only `e2e/**` / `tools/e2e.sh`: the e2e jobs. - Desktop job only when `app/`, `crates/`, `Cargo.*`, `devserver/`, `ci/`, the smoke script or the workflow changed. - A branch is compared with its merge base on main (the whole branch, not the last commit). - **On main**: a merge commit whose tree equals its second parent's (the PR head) was tested as-is, and `pr-merge` only merges when all its runs passed; confirmed through the API, else everything runs. Otherwise the merge's whole diff is classified as above. - `workflow_dispatch`: everything. New: a **nightly full run on main**, so nothing skipped stays untested for more than a day. 3. Later, measured first: cache `~/.cargo/registry` and build output on the shared runners (each job downloads every crate today). Expected: a code PR ~17 min on the branch and ~1 min on main (most of the time); a docs/roadmap PR ~1–2 min each. CLAUDE.md's workflow section is updated to match (it also says main's run includes the soak test; that's the nightly `soak` workflow, while the e2e run has the short soak spec).

littlefox added the

area/ci

kind/infra

labels

2026-10-27 18:06:11 +00:00

littlefox referenced this issue from a commit

2026-10-27 21:50:04 +00:00

CI: run what a change needs, and the browser profiles in parallel

littlefox referenced this issue from a commit

2026-10-27 22:19:53 +00:00

CI: each browser job builds the web app itself

littlefox referenced this issue from a pull request that will close it,

2026-10-27 22:45:07 +00:00

CI: run what a change needs, and the browser profiles in parallel #366

littlefox closed this issue

2026-10-27 22:45:42 +00:00

littlefox referenced this issue from a commit

2026-10-27 22:45:43 +00:00

Decision: CI runs what a change needs

littlefox commented

2026-10-27 22:49:10 +00:00

Author

Owner

Merged in #366; main green (run #649).

Verified:

tools/test_ci_plan.py: 18 tests (rules, whole-branch diffs, merges of up-to-date and behind branches, cancelled main runs).
Run #636 (all jobs, separate build job): 27.9 min. Run #639 (each browser job builds itself): 24.4 min, with five other runs competing for runners. Before: 36-40 min.
Run #649, main's run of the merge: the plan compared it with the PR head 0c6c2ef7 (0 files differ) and ran nothing: 2.5 min instead of ~40.
The job token can read the runs API; without an answer, everything runs.

Merged in #366; main green (run #649). Verified: - `tools/test_ci_plan.py`: 18 tests (rules, whole-branch diffs, merges of up-to-date and behind branches, cancelled main runs). - Run #636 (all jobs, separate build job): 27.9 min. Run #639 (each browser job builds itself): 24.4 min, with five other runs competing for runners. Before: 36-40 min. - Run #649, main's run of the merge: the plan compared it with the PR head 0c6c2ef7 (0 files differ) and ran nothing: **2.5 min** instead of ~40. - The job token can read the runs API; without an answer, everything runs.

littlefox referenced this issue

2026-10-28 02:26:19 +00:00

Roadmap: 0.8 done #379

Rows
Columns

CI: run what a change needs, and the browser profiles in parallel #358

Where the time goes (run #619, typical)

Proposal