1 CI and Builds
mitosis edited this page 2026-07-29 08:11:35 -05:00

Every build on this platform runs on one CI pool: ephemeral, scale-from-zero, and non-privileged by construction. This page covers what runs-on: isolated gives you, how images actually get built without a Docker daemon, and the one bootstrap step a fresh platform needs.

One pool: runs-on: isolated

Declare runs-on: isolated in a workflow and you get an ephemeral runner with a full toolchain, a rootless image builder, and a smoke Postgres — and zero ability to escape to the node or other tenants. It is the only pool: runs-on: docker is never claimed by anything, because the privileged docker-in-docker pool was deleted outright (it exposed DOCKER_HOSTdocker run --privileged -v /:/host → node root).

Each runner pod is shaped for isolation:

  • automountServiceAccountToken: false — tenant code gets no kube API token.
  • The main container is the plat/ci-runner toolchain image (bun, node, git, curl baked in), running forgejo-runner one-job — one job, then gone. The pool registers ephemeral runners from a global instance-scoped secret, so every repo can target isolated.
  • Execution label is isolated:host — steps run directly in the toolchain container. No job container, no implicit checkout: each job clones its exact commit itself using the ephemeral, repo-scoped github.token.
  • Sidecars: a rootless BuildKit builder (moby/buildkit rootless, uid 1000, not privileged, reached over a UNIX socket), a postgres:16 smoke database at 127.0.0.1:5432 (creds ci/ci, db ci), and a CA-bundle init so pushes to the platform's HTTPS registry endpoint verify.
  • An egress-only NetworkPolicy allows exactly: DNS, the Forgejo namespace (clone + registry pull), the edge (HTTPS push), and the MCP provisioning endpoints (/ci/preview, /ci/sync-env, which authorize by probing the caller's Actions token for push rights on that repo). Public internet egress is open except RFC1918 ranges and the cloud metadata IP 169.254.169.254.

Scale-from-zero

The pool is a KEDA ScaledJob (0 → 6 replicas, 15s polling, 40-minute job deadline). The trigger is a postgresql query against Forgejo's own database: SELECT COUNT(*) FROM action_run_job WHERE status = 5 AND runs_on::text LIKE '%isolated%'. Waiting isolated jobs spawn runner pods; no waiting jobs, no pods, no idle cost.

The build pipeline

App workflows (see the five that ship in _app-template: check, guard-ppt, preview, release, sync-env) build and ship like this:

  1. Buildbuildctl --addr "$BUILDKIT_HOST" drives the rootless BuildKit sidecar. Images push to the platform's public HTTPS registry endpoint (buildctl won't honor plain-HTTP push — that's what the CA bundle is for).
  2. Signcosign sign by digest with the platform key (--tlog-upload=false, self-hosted; skipped gracefully when no key is configured — the Kyverno verifyImages policy is Audit-first, so a keyless fresh platform still deploys).
  3. Charthelm package charts/app --version "$V" --app-version "$V" (the chart is renamed to the app), then helm push to oci://<registry>/<org>/charts.

Previews use tag 0.0.0-pr.<N>.<sha7> — sha-suffixed so every push is a distinct chart+image version Flux actually rolls on. Releases are v* git-tag pushes; the bare semver becomes image tag, chart version, and APP_VERSION. From there Flux takes over — see Preview Environments and Platform Internals.

Credentials are split by design: github.token (ephemeral, scoped to this repo) handles clone, API reads, and provisioning calls; REGISTRY_TOKEN is registry auth only — a per-org write:package-only credential, never an admin token.

Writing workflows: there is no Docker daemon

No node on the platform runs a Docker daemon for CI, anywhere. For a workflow author that means:

  • No docker run, no docker build, no service containers. Your tools are bun (install, test, build), buildctl (images), helm (charts), cosign (signing), plus git and curl.
  • Need a database for tests? It's already there: the Postgres sidecar at 127.0.0.1:5432. The template's check workflow uses it to lint, typecheck, build, then boot the real server and smoke-assert its endpoints — all before any image ships.
  • Need to hit the platform API? The in-cluster registry is forgejo-http.forgejo.svc.cluster.local:3000; the provisioning API is http://mcp.mcp.svc.cluster.local.

guard-ppt

guard-ppt.yml runs on every push and PR and fails if any workflow file uses the pull_request_target trigger — the one Actions trigger that hands base-repo secrets to untrusted PR code. It's wired as a required status check on main for plat/mitosis and plat/_app-template, so reintroducing it is un-mergeable there; making it required on every app repo is a known open follow-up. The rest of the workflow attack surface and its controls are covered in Security Model.

The bootstrap chicken-and-egg

The runner pool's own container image is plat/ci-runner — and on a fresh platform, nothing can build it, because building anything needs a runner. Germination breaks the cycle off-cluster: it docker builds the seeded ci-runner repo on the host as git.<domain>/plat/ci-runner:<the tag pinned in gitops> (0.1.0 today — germinate greps the tag from the gitops pin so it can't skew) and k3d image imports it into the node. On a bare VPS with no Docker on the box, it's a one-time docker build + docker save | ssh ... ctr images import from any Docker machine — the tag must match the gitops pin, and the exact block is in Grow Your Own Platform. Skip it and the platform stands up fine but no CI ever runs. Once the pool is alive, the ci-runner repo's own release workflow rebuilds the image through normal CI, and the loop is closed.