The engine room: how one git repo becomes a running platform. This page walks plat/gitops top to bottom — the Flux graph, secrets, identities, TLS, the registry, and tenant governance — so you can trust the machinery by reading it. For the guided overview first, see Architecture.

The gitops repo

Everything the platform runs — from Forgejo itself to every tenant app — is a file in plat/gitops. Changing the platform is a git commit; Flux converges the cluster to the repo on a 1-minute interval.

The repo root holds .sops.yaml and clusters/. Under clusters/local/:

• platform/ — the platform components (one directory each)
• apps/ — per-org, per-app overlays (see below)
• domains.yaml — bring-your-own-domain registry: one line <domain>: <org>, and apps in that org resolve to it
• flux-kustomizations.yaml — the Flux entrypoint: 12 Kustomizations with an explicit dependsOn DAG

The Flux graph

Compactly, with ← meaning "depends on":

• cert-manager — root; everything TLS descends from it
• pki ← cert-manager
• monitoring ← cert-manager; monitoring-scrapes ← monitoring
• kyverno ← cert-manager; kyverno-policies ← kyverno
• flow ← cert-manager, traefik
• platform ← cert-manager, traefik, keda
• apps ← platform, pki
• traefik, keda, cnpg-operator — independent roots

Two design decisions are written into the file as comments: monitoring is deliberately not a dependency of apps ("an observability hiccup must never stall application reconciliation"), and ServiceMonitors live in the separate monitoring-scrapes Kustomization because a dry-run failure on an unknown kind blocks an entire Kustomization. SOPS decryption is declared on exactly four Kustomizations: monitoring, flow, platform, apps.

What each component is:

Component	One line
forgejo	The control plane itself: Forgejo Helm chart + CNPG Postgres, the git.espoautos.com Ingress, the isolated CI runner pool
traefik	Edge ingress; system-cluster-critical priority with guaranteed CPU so a saturated node can't kill the edge
cnpg-operator	CloudNativePG — runs every Postgres on the platform (Forgejo's and each app's)
keda	Powers the scale-from-zero CI pool via a ScaledJob postgres trigger
mcp	The platform-bot / control-plane API at mcp.espoautos.com
agents	The AI build-crew dispatcher, with all token-flow tuning as env
flow	The Flow Deck console (read-only surface)
monitoring	kube-prometheus-stack: Prometheus (60d retention), Grafana with Forgejo OAuth
monitoring-scrapes	ServiceMonitors for agents and mcp /metrics
kyverno / kyverno-policies	Admission engine + the 12-policy governance baseline (see below)
letsencrypt	Public ACME (DNS-01) ClusterIssuer + public wildcard certificate (currently on the ACME staging directory — a known open hardening item)
cosign-signing	The platform's image-signing keypair — source of provenance trust
registry-node-config	The node-local registry endpoint on 127.0.0.1:31100 (see Registry)
node-fabric	Per-node DaemonSet keeping tx-checksum offload disabled on flannel's vxlan device (multi-node packet integrity)
plat-storage	In-cluster MinIO + a cnpg-backups bucket with a least-privilege backup user for Postgres WAL backups
plat-system	The reserved-subdomain registry (git, mcp, grafana, console, ...)
pki	The private platform CA (see TLS)
coredns	Split-horizon DNS: platform hostnames resolve to traefik from inside the cluster
cert-manager	Issues every certificate; root of the DAG

On a freshly-germinated lean daughter, monitoring, monitoring-scrapes, and flow are stripped at germination — observability is parent-only on a lean daughter today — and everything else above is present from birth.

How an app lands — and leaves

Every app environment is an overlay at clusters/local/apps/<org>/<app>/<env>/, where <env> is prod, pr-N (a PR preview), or occasionally dev. An overlay is live iff it is listed in apps/kustomization.yaml — the list, not the directory tree, is the deploy roster.

Each env dir is a kustomization.yaml, a manifests.yaml, and four sealed secrets. manifests.yaml renders the whole app:

1. Namespace <org>--<app>--<env> with plat.sh/* ownership labels and pod-security.kubernetes.io/enforce: restricted
2. CNPG Postgres <app>-pg (only when the app declares a database)
3. Certificate wildcard-plat-tls for <app>-<org>.espoautos.com
4. OCIRepository pinning a chart tag from oci://forgejo-http.forgejo.svc.cluster.local:3000/<org>/charts/<app> (previews pin a PR tag like 0.0.0-pr.13.0757a36)
5. HelmRelease wiring the DB secret and self-serve env into the pod
6. Ingress on <app>-<org>.espoautos.com (previews: pr-N-<app>-<org>.espoautos.com)

Nobody writes this YAML by hand: create_app and preview provisioning render the overlay, seal the secrets, append the roster entry, and commit. Flux applies it within a minute.

• Rollout: a release bumps the chart tag in the overlay → the OCIRepository sees the new tag → the HelmRelease upgrades.
• Config change: a self-serve secret change bumps a userEnvRev hash → pod annotation changes → the Deployment re-rolls. (Chart env is read only at pod start.)
• Decommission: remove the roster entry and the overlay dir. Flux prune: true deletes everything, namespace included. Deleting only the namespace does not work — Flux resurrects it. The overlay is the source of truth.

Secrets: SOPS + age

Every secret in the repo is sealed with SOPS to a single age recipient — the platform's sovereign key. Two rules in .sops.yaml:

1. clusters/.+\.yaml$ with encrypted_regex: ^(data|stringData)$ — only data fields are sealed; metadata and comments stay readable in git
2. clusters/.+/values\.env$ — whole file

The age private key lives cluster-side in the sops-age secret in flux-system; only the four Kustomizations that declare decryption can use it. When a platform reproduces, the daughter mints a fresh key and every secret is regenerated and resealed — the parent's sealed values in the seed are inert ciphertext (see Grow Your Own Platform).

The four sealed secrets in every app overlay: the image pull secret, the app's Forgejo OAuth credentials, a mirror of the platform CA cert, and the app's self-serve env — which is sealed from the repo's Actions secrets by the sync-env flow, never edited in gitops directly (see Secrets and Config).

Identities: plat, ci-bot, registry bots

• plat — the site-admin machine user. Owns the plat/* system repos; its PAT is what platform services authenticate with. Humans don't work as plat.
• ci-bot — a restricted user whose token is write:repository, write:issue only; it opens automated PRs (e.g. seed refreshes).
• Registry credentials — per-org, write:package-only. The REGISTRY_TOKEN an app's CI pushes images and charts with is one of these — never an admin PAT. Legacy repos are still being rotated off a shared registry bot.

Power is scoped by construction, not policy documents. The human-facing side of this split is in Users Teams and Access; the adversarial analysis is in Security Model.

TLS: two realms

• Private platform CA (pki/): a self-signed bootstrap issuer creates a 10-year plat-local-ca CA (secret plat-local-ca-tls), and the plat-local-ca-issuer signs every internal *.espoautos.com certificate — including each app's wildcard-plat-tls.
• Public (letsencrypt/): the letsencrypt-dns01 ClusterIssuer (ACME DNS-01 via Cloudflare) issues wildcard-public-tls for browser-trusted hosts (currently on the ACME staging directory — a known open hardening item).

On a local platform, trust the CA once and every app is green-padlocked: export plat-local-ca-tls and add it to your OS trust store (security add-trusted-cert on macOS, update-ca-certificates on Linux) — the YOUR PLATFORM card prints the exact command for your OS at germination.

Registry and the node mirror

Every app's chart and image live in Forgejo's own package registry, under the owning org: charts at oci://forgejo-http.forgejo.svc.cluster.local:3000/<org>/charts/<app>, images at git.espoautos.com/<org>/<app>. Platform components use plat/charts/* the same way — the platform ships itself through the same registry it gives tenants.

Nodes pull platform images through a local mirror: containerd maps git.espoautos.com → http://127.0.0.1:31100 via a registries.yaml frozen into each node at cluster creation. At runtime, a hostNetwork Caddy DaemonSet binds loopback-only 127.0.0.1:31100 on every node and forwards to a small rewriter Deployment that rewrites the registry's Www-Authenticate realm to the local endpoint — so containerd's token fetch resolves locally, with no public DNS required, on any domain. This is why platform HelmReleases carry imagePullSecrets: []: registry auth is node-level.

Two more pieces:

• A docker.io pull-through cache (registry v3 proxying mirror.gcr.io, 10Gi cache) keeps CI base-image pulls off Docker Hub's rate limits.
• Provenance: CI cosign-signs every app image, and the verify-image-provenance ClusterPolicy admits only images that validate against the platform's public key. The keypair lives in cosign-signing/, deliberately under the platform Kustomization — the one with SOPS decryption.

Kyverno: tenant governance

Twelve ClusterPolicies form the governance baseline, and the Enforce flip is active (HA admission controller, fail-closed-bounded). The two that shape every tenant namespace:

• plat-ns-resource-governance — generates a ResourceQuota and LimitRange in every app namespace
• plat-ns-network-governance — generates default-deny plus allow NetworkPolicies in every app namespace

Both key off the plat.sh/app namespace label with synchronize: true self-heal: delete a generated quota and it comes back. The rest of the roster: disallow-privileged-hostpath, require-resource-limits, restrict-image-registries, plat-ns-dos-bound, disallow-unmanaged-secrets, require-registered-ingress-host, verify-image-provenance, no-admin-clusterrole-binding, agent-runs-use-scoped-role, and require-forwardauth-on-app-ingress (audit-only; its Enforce component is deliberately not shipped).

The admission webhook is scoped to namespaces where plat.sh/app exists — tenant namespaces only. Break-glass exists and is loud: label a namespace plat.sh/break-glass=true to exempt it from admission while background policies keep reporting.

When something's weird

• flux get kustomizations — which of the 12 is unhappy, and why. Reconciliation is 1-minute; most "why hasn't it shipped" questions answer themselves here.
• kubectl -n agents logs deploy/agents — the build dispatcher; per-run crew logs live in the pod under /data/logs.
• The repo's Actions tab — every CI build's logs.
• An app namespace you deleted came back? That's Flux doing its job — decommission through the overlay.
• An Ingress that never matches? Every public Ingress must declare entrypoints web,websecure — tunnel-fronted traffic arrives on web, so a websecure-only router silently matches nothing.