neokom.lysator.liu.se down after the 0.5.0 autoroll: caddy "Operation not permitted" #268

New issue

Closed

opened 2026-10-22 05:33:09 +00:00 by littlefox · 1 comment

littlefox commented

2026-10-22 05:33:09 +00:00

Owner

Reported by the user, 2026-10-22: neokom.lysator.liu.se unreachable since the autoroll (503 from the gateway, "upstream connect error … connection timeout"). The pod log: /usr/local/bin/neokom-web: line 74: /usr/local/bin/caddy: Operation not permitted.

Cause: 0.5.0 was the first image built in CI (#201), and that builder kept the file capability (cap_net_bind_service) the caddy image puts on its binary; 0.4.0, built locally, had lost it. Lysator runs the pod with all capabilities dropped and allowPrivilegeEscalation: false, and a binary with a file capability can't be executed there at all. Reproduced with docker run --user 999:999 --read-only --cap-drop ALL --security-opt no-new-privileges.

Fix (hotfix 0.5.1, from v0.5.0): copy caddy without the capability (it listens on 8080 and needs none); the image's CI check runs it locked down the same way.

Reported by the user, 2026-10-22: neokom.lysator.liu.se unreachable since the autoroll (503 from the gateway, "upstream connect error … connection timeout"). The pod log: `/usr/local/bin/neokom-web: line 74: /usr/local/bin/caddy: Operation not permitted`. Cause: 0.5.0 was the first image built in CI (#201), and that builder kept the file capability (`cap_net_bind_service`) the caddy image puts on its binary; 0.4.0, built locally, had lost it. Lysator runs the pod with all capabilities dropped and `allowPrivilegeEscalation: false`, and a binary with a file capability can't be executed there at all. Reproduced with `docker run --user 999:999 --read-only --cap-drop ALL --security-opt no-new-privileges`. Fix (hotfix 0.5.1, from v0.5.0): copy caddy without the capability (it listens on 8080 and needs none); the image's CI check runs it locked down the same way.

littlefox added this to the 0.6: Performance milestone

2026-10-22 05:33:09 +00:00

littlefox added the

kind/bug

area/packaging

area/web

labels

2026-10-22 05:33:09 +00:00

littlefox referenced this issue from a pull request that will close it,

2026-10-22 05:33:10 +00:00

Hotfix 0.5.1 into main: the web image runs with capabilities dropped #269

littlefox commented

2026-10-22 06:04:41 +00:00

Author

Owner

Hotfix 0.5.1 tagged from v0.5.0 (ceb496c). web-image run #445 built it, ran it locked down (user 999, read-only, all capabilities dropped, no new privileges) next to a lyskomd, and pushed 0.5.1 and latest. The autoroll rolled the pod about 3 minutes later. Checked from outside: https://neokom.lysator.liu.se/ answers 200 with the app, and /relay upgrades to a WebSocket (101). PR #269 brings the fix into main; this closes when it merges.

Hotfix 0.5.1 tagged from v0.5.0 (ceb496c). web-image run #445 built it, ran it locked down (user 999, read-only, all capabilities dropped, no new privileges) next to a lyskomd, and pushed 0.5.1 and latest. The autoroll rolled the pod about 3 minutes later. Checked from outside: https://neokom.lysator.liu.se/ answers 200 with the app, and /relay upgrades to a WebSocket (101). PR #269 brings the fix into main; this closes when it merges.

littlefox closed this issue

2026-10-22 06:17:25 +00:00