autoroll

A deliberately tiny in-cluster watcher that rolls Kubernetes Deployments when the mutable :latest tag behind their container image repoints to a new digest in a Forgejo registry. Kubernetes never notices a mutable tag moving on its own; autoroll closes that gap without ArgoCD Image Updater or Keel, and without ever patching a Deployment.

How it works (design rationale)

The cluster is the state. autoroll keeps no database, no files, no CRDs. Every tick it compares two things it reads live: the digest the registry returns for a container's tag (Docker-Content-Digest) against the digest the pods are actually running (pod.status.containerStatuses[].imageID). If they differ, the running pod is stale. There is nothing to persist because both sources of truth already live in the cluster and the registry.

The only write is pod deletion. autoroll never patches, restarts, or otherwise mutates a Deployment — it deletes at most one stale pod per Deployment per tick and lets the ReplicaSet recreate it against the new image. That single-write design is what keeps the tool invisible to ArgoCD self-heal: the Deployment object ArgoCD owns is never touched, so enrolled apps stay Synced throughout a rollout.

One pod at a time, and only when settled. A stale pod is deleted only while its Deployment is settled: no pod terminating, every pod Ready, and at least spec.replicas ready pods. The replacement must become Ready before the next stale pod goes, which gives a real rolling update for replicas > 1 however often ticks fire (e.g. a burst of webhooks). Only pods whose container image string equals the Deployment template's are compared, so pods left over from an older template during an ArgoCD rollout are left to the Deployment controller. Succeeded/Failed pods (e.g. evicted) are ignored. Note that a Deployment with a persistently unready pod never rolls — autoroll logs that it is waiting and does nothing, which is the safe direction.

Failure is never "changed". Any registry error — network, 401, 404, a missing header — is treated as unknown and skips that image for the tick; it never reads as "the digest changed". A pod still pulling (no resolved imageID) is skipped, not deleted. A newly rolled image that crash-loops already matches the registry digest, so autoroll never deletes it again, and because that pod never becomes Ready, the rollout halts there instead of spreading the broken image to the remaining replicas. The multi-arch case is handled explicitly: the manifest HEAD offers all index/manifest-list media types so the registry returns the index digest that containerd reports as imageID — the number-one way a naive watcher ends up deleting healthy pods forever.

Enrollment

Two things enroll an app (see examples/example-app.yaml):

1. Label aethnet.se/autoroll: "true" on the Deployment object (not the pod template — so enrolling doesn't itself trigger a rollout).
2. imagePullPolicy: Always on every enrolled container using a mutable tag. Without it a recreated pod could reuse the node's cached image and come back stale. To stay loop-safe, autoroll skips any tag-referenced container whose pull policy isn't Always — it logs a loud warning each tick and never deletes that container's pods (so it won't force a restart it would only have to redo). Set imagePullPolicy: Always to opt the container into rollout.

Digest-pinned images (…@sha256:…) and images not hosted on FORGEJO_HOST are ignored.

Configuration (env vars only)

Var	Default	Notes
FORGEJO_HOST	— (required)	bare host[:port], e.g. forgejo.aethnet.se; also the image-host filter
FORGEJO_USER	— (required)	Forgejo user that owns FORGEJO_TOKEN
FORGEJO_TOKEN	— (required)	Forgejo token, read:package scope; sent with FORGEJO_USER as HTTP Basic auth
WEBHOOK_SECRET	— (required)	shared secret for webhook HMAC verification
POLL_INTERVAL_SECONDS	120
LABEL_SELECTOR	aethnet.se/autoroll=true
LISTEN_ADDR	0.0.0.0:8080
DRY_RUN	false	log intended deletions without deleting
RUST_LOG	info	tracing env filter

HTTP endpoints

• GET /healthz → 200. Used for liveness/readiness probes.
• POST /webhook → verifies the Forgejo X-Forgejo-Signature (HMAC-SHA256 over the raw body, constant-time compare against WEBHOOK_SECRET) and, on success, nudges an immediate reconcile and returns 202. Bad or missing signature → 403, no reconcile. The payload is never parsed or trusted — a valid webhook simply runs the same full reconcile the poll timer runs, so the worst an attacker with the secret can do is cause an extra no-op tick.

Poll timer and webhook both nudge a single reconcile worker through one Notify; ticks never overlap and concurrent nudges coalesce into at most one follow-up tick.

RBAC

Cluster-scoped and minimal (deploy/clusterrole.yaml): deployments → list; pods → list, delete. Bound to a dedicated autoroll ServiceAccount in the autoroll namespace. pods/delete is the only write verb the tool holds.

Deploy

# 1. Create the namespace, SA, and RBAC.
kubectl apply -f deploy/namespace.yaml
kubectl apply -f deploy/serviceaccount.yaml
kubectl apply -f deploy/clusterrole.yaml
kubectl apply -f deploy/clusterrolebinding.yaml

# 2. Create the Secret (do NOT commit real values — examples/secret.example.yaml
#    is a stub).
kubectl -n autoroll create secret generic autoroll \
  --from-literal=FORGEJO_TOKEN=... \
  --from-literal=WEBHOOK_SECRET=...

# 3. Edit deploy/deployment.yaml (set FORGEJO_HOST + image), then:
kubectl apply -f deploy/deployment.yaml
kubectl apply -f deploy/service.yaml

Point a Forgejo package-push webhook at POST http(s)://<autoroll>/webhook with the same WEBHOOK_SECRET to trigger reconciles on push instead of waiting for the poll.

Build

cargo test          # unit tests for the pure logic
cargo build --release
docker build -t autoroll:latest .   # static musl binary → scratch image (CA roots are compiled in)
docker build --target test .        # run the tests inside the build image

CI (.forgejo/workflows/build-push.yml) runs the test stage on every push to main, then pushes :latest and :<sha> to the registry of whichever Forgejo instance runs it (the host of its public ROOT_URL, or the Actions variable REGISTRY if set), at <host>/<owner>/<repo> lowercased — e.g. forgejo.serana.aethnet.se/aetherius/autoroll or forge.lysator.liu.se/<owner>/autoroll. The deployed autoroll is enrolled in itself, so the new :latest rolls it. Docs-only pushes are skipped.

Per instance it needs the Forgejo Actions secrets REGISTRY_USER and REGISTRY_TOKEN (a write:package token on that instance), and a runner with the docker-builder label — both k8s-aeth and k8s-lys provide one.

Manual acceptance checklist (requires a cluster)

The pure logic is unit-tested (cargo test); the following are verified by hand against a real cluster:

• DRY_RUN=true run logs correct intended deletions and deletes nothing.
• Pushing a new :latest to an enrolled app → its pod is replaced within one poll interval.
• Forgejo webhook configured → pod replaced within seconds of a push.
• Bad/missing webhook signature → 403, no reconcile.
• Registry taken offline → warnings only, no deletions.
• Enrolled ArgoCD apps stay Synced throughout a rollout.