Packaging: nix job hung after the build finished #119

Closed
opened 2026-10-18 17:40:31 +00:00 by littlefox · 3 comments
Owner

CI run #196 (packaging, PR #116, 6e3c9139): the nix job compiled neokom ('Finished release profile in 8m 25s', 15:40), printed 'Executing cargoInstallPostBuildHook' and then nothing until the job stopped at 15:53 as failed. The same job passed on #191 (same PR, before main was merged in) and on #194 (v0.2.0).

Find out what the install hook waits on (store write, runner disk, a timeout in the workflow), and whether the job needs a longer timeout or a retry.

CI run #196 (packaging, PR #116, 6e3c9139): the nix job compiled neokom ('Finished release profile in 8m 25s', 15:40), printed 'Executing cargoInstallPostBuildHook' and then nothing until the job stopped at 15:53 as failed. The same job passed on #191 (same PR, before main was merged in) and on #194 (v0.2.0). Find out what the install hook waits on (store write, runner disk, a timeout in the workflow), and whether the job needs a longer timeout or a retry.
Author
Owner

Looked at this again (2026-10-19). The nix job has passed in every packaging run since #196: #203, #220, #230, #238 and #240 (v0.3.0), and about ten before it. Each took 20 to 27 min. #196 ran 29 min and ended right after the build, with the arch, debian and fedora jobs running at the same time. That fits a runner that was overloaded or killed (memory, disk) rather than anything in the flake. The job log can't be fetched any more.

Leaving it open, unchanged, as a watch item. If it happens again, save the job log and the runner's dmesg/disk state before re-running, and then decide between a timeout-minutes with one retry of the build step and running the nix job after the other three instead of next to them.

Looked at this again (2026-10-19). The nix job has passed in every packaging run since #196: #203, #220, #230, #238 and #240 (v0.3.0), and about ten before it. Each took 20 to 27 min. #196 ran 29 min and ended right after the build, with the arch, debian and fedora jobs running at the same time. That fits a runner that was overloaded or killed (memory, disk) rather than anything in the flake. The job log can't be fetched any more. Leaving it open, unchanged, as a watch item. If it happens again, save the job log and the runner's dmesg/disk state before re-running, and then decide between a timeout-minutes with one retry of the build step and running the nix job after the other three instead of next to them.
Author
Owner

Again on 2026-10-21, packaging run #397 (PR #247, 935fed70): arch, debian and fedora passed (16, 28 and 32 min, running at the same time), and nix failed after 35m32s with no steps recorded: the web UI has no step list and no log for the job, and there is no logs-nix artifact. The same commit's ci run passed. So the job died, or was killed, before or outside its steps. Again it fits an overloaded runner.

Re-running with an empty commit, since my token can't re-run jobs. Moved to 0.6 because it blocks merges now. Next step if it repeats: run nix after the other three (needs: [arch, debian, fedora]) so the four builds don't compete for the runner.

Again on 2026-10-21, packaging run #397 (PR #247, 935fed70): arch, debian and fedora passed (16, 28 and 32 min, running at the same time), and nix failed after 35m32s with **no steps recorded**: the web UI has no step list and no log for the job, and there is no logs-nix artifact. The same commit's ci run passed. So the job died, or was killed, before or outside its steps. Again it fits an overloaded runner. Re-running with an empty commit, since my token can't re-run jobs. Moved to 0.6 because it blocks merges now. Next step if it repeats: run nix after the other three (`needs: [arch, debian, fedora]`) so the four builds don't compete for the runner.
Author
Owner

Merged in #274 (main #462 green). The nix job now runs after arch, debian and fedora instead of next to them. Verified on packaging run #459 (the PR's own): arch 09:15–09:28, fedora 09:25–09:45, debian 09:30–09:36, then nix 09:49–10:05, all green. The release job still waits for all four.

Merged in #274 (main #462 green). The nix job now runs after arch, debian and fedora instead of next to them. Verified on packaging run #459 (the PR's own): arch 09:15–09:28, fedora 09:25–09:45, debian 09:30–09:36, then nix 09:49–10:05, all green. The release job still waits for all four.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
littlefox/neokom#119
No description provided.