Installation

The installation is performed in two phases:

Infrastructure deployment with Ansible for gateway and K3s cluster setup
Applications deployment with ArgoCD following a GitOps approach

TIP

Already have a cluster? Phase 1 is optional. If you run managed Kubernetes (EKS/GKE/AKS/DOKS/...) or any existing cluster, skip straight to Applications (GitOps) — the argo-cd/ tree only needs a working kubeconfig and does not depend on Ansible, K3s, or the gateway host. See Using an existing cluster for what to adjust.

Prerequisites

Following tools need to be installed on the computer running the deployment:

ansible - infrastructure as code software tools.
age - simple, modern and secure encryption tool (only for the optional Sops secrets).
helm - Kubernetes package manager.
htpasswd - bcrypt password hashing (apache2-utils on Linux, ships with macOS).
kubectl - Kubernetes command-line tool.
sops - simple and flexible tool for managing secrets (only for the optional Sops secrets).
sshpass - non-interactive ssh password auth.
yq - portable command-line YAML, JSON, XML, CSV, TOML and properties processor.

# Clone the repository
git clone --depth 1 https://github.com/this-is-tobi/ohmlab.git && cd ./ohmlab && rm -rf ./.git

# Copy inventory example to inventory
cp -R ./ansible/inventory-example ./ansible/inventory

Ansible Vault

Sensitive values (passwords, tokens, certificates) are stored in ansible/inventory/group_vars/vault.yml and encrypted at rest with ansible-vault.

Create a vault password file (never committed — already in .gitignore):

# Generate a strong random password
openssl rand -base64 32 > ./ansible/.vault_password

Fill in the secrets in ansible/inventory/group_vars/vault.yml:

yaml

vault_ansible_password: <ssh-password>
vault_pihole_password: "" # auto-generated if empty
vault_wireguard_password: "" # auto-generated if empty
vault_k3s_token: "" # populated during cluster bootstrap
vault_k3s_ca_data: "" # populated during cluster bootstrap

Encrypt the vault file:

cd ansible && ansible-vault encrypt inventory/group_vars/vault.yml

Notes:
To edit secrets later: ansible-vault edit inventory/group_vars/vault.yml
Auto-generated values are created once and persisted in gitignored files next to the inventory — inventory/group_vars/.pihole_password, .wireguard_password (by the gateway roles) and .pi_password (by scripts/setup-pi.sh) — and reused on later runs. Copy them into the vault when you want them under version control.
The vault password file path is configured in ansible.cfg (vault_password_file = .vault_password).

Notes:
PiHole and Wireguard installation can be ignored by setting enabled: false in gateway group_vars.

Settings

Infrastructure

Update the hosts file and group_vars files to provide the appropriate infrastructure settings.

All sensitive values are indirected through vault_* variables in group_vars/vault.yml (see Ansible Vault above). Non-secret settings are in all.yml, gateway.yml and k3s.yml.

To create admin access to the machines, provide admin user information in group_vars/all.yml (adminUsers). Each entry creates:

a dedicated OS account on every host with the given sshPubKey in its own authorized_keys (attributable logins — keys are never appended to the shared ansible_user account), plus optional passwordless sudo with sudo: true;
a Kubernetes user (x509 client certificate + kubeconfig) with the RBAC tier set by role: namespace-admin (default — a personal namespace with full rights inside it), cluster-admin, or view.

state: absent removes the OS account, the SSH key and all Kubernetes RBAC. Note that the issued client certificate keeps authenticating until it expires (Kubernetes has no CRL) — deleting the bindings is what revokes authorization. Certificates are valid one year and re-issued automatically when a run finds them within 30 days of expiry.

Applications (GitOps)

Applications are managed by a two-level ApplicationSet hierarchy:

The root manager ApplicationSet (shipped by ohmlab) discovers every folder under ./argo-cd/instances/ and emits one Application per instance pointing at the ./argo-cd/apps/instance-manager chart.
That chart in turn renders two child ApplicationSets per instance — core-<instance> and tenant-<instance> — that fan out into the actual leaf Applications.

Configuration is split per instance and per scope:

Path	Purpose
./argo-cd/instances/<instance>/instance.yaml	Per-instance metadata: cluster destination, env, repos, AppProject bindings.
./argo-cd/instances/<instance>/core.yaml	Core tier app catalog (platform/infra/identity/observability/security).
./argo-cd/instances/<instance>/tenant.yaml	Tenant tier app catalog (user-facing services).
./argo-cd/instances/<instance>/values/core/<app>.yaml	Per-instance Helm values for each core app.
./argo-cd/instances/<instance>/values/tenant/<app>.yaml	Per-instance Helm values for each tenant app.
./argo-cd/instances/<instance>/values/core/ohmlab.yaml	Bootstrap values for core ArgoCD + the root `manager` AppSet + the `admin-core` / `admin-tenant` AppProjects.
./argo-cd/apps/<app>/	Helm chart catalog (chart sources only — values live in the trees above).

To enable or disable a service for an instance, edit the matching core.yaml or tenant.yaml and flip the "enabled" field on the relevant entry.

Per-app overrides supported in the JSON catalogues (all optional):

Field	Default	Use case
`chart`	same as `app`	Use a different chart directory under `argo-cd/apps/`.
`chartPath`	`argo-cd/apps/<chart>`	Point at a chart outside `argo-cd/apps/` (e.g. self-managed `ohmlab`).
`catalog`	`defaultCatalog`, else `git` (the repository of the instance)	Take the chart from another catalog, see Catalogs and Charts from a registry.
`releaseName`	same as `app`	Adopt an existing helm release for self-management.
`namespace`	`<prefix><app><suffix>`	Pin to an explicit namespace (e.g. `argocd-system`).
`destination.server`	`instance.yaml.destination.server`	Target a different cluster (multi-cluster).
`valuesPath`	`argo-cd/instances/<instance>/values/<scope>/<app>.yaml`	Point to a non-conventional values file.
`targetRevision`	the version of its catalog, else `instance.yaml.targetRevision`	Pin the app to a git revision, to another catalog version, or to a chart version with a registry catalog.
`syncWave`	required	ArgoCD sync ordering.

Catalogs

Where the charts come from is the instance's choice, declared once in instance.yaml. Without any setting every chart is the folder argo-cd/apps/<chart> of repoURL at targetRevision, the catalog named git. A bundle catalog gives the same folder from another repository (git, or an oci:// artifact holding the same layout) at one version, so an instance runs one tested set of charts and moves all of them by changing one line:

yaml

# argo-cd/instances/<instance>/instance.yaml
catalogs:
  release:
    repoURL: https://github.com/this-is-tobi/ohmlab.git
    version: v0.1.0
defaultCatalog: release # every entry without `catalog:` comes from it

# argo-cd/instances/<instance>/core.yaml
- app: gitea              # runs `release` at v0.1.0
- app: kyverno
  targetRevision: v0.0.9  # this app stays on an older version
- app: my-own-app
  catalog: git            # a chart of the instance repository, as without a default

catalogs.git is reserved. defaultCatalog must be declared under catalogs (or be git), and a bundle needs a repoURL: the manager chart refuses to render otherwise, instead of Applications quietly coming from somewhere else.
A bundle without version leaves the revision to each entry's targetRevision; an entry with neither gets an invalid revision, so nothing resolves by accident. A catalog name that is not declared gives the Application no repository.
The AppProject of the tier must list the repository of every catalog in sourceRepos, spelled exactly like the catalog (with the oci:// scheme for an artifact).
The ohmlab entry that ./run.sh -b installs needs a registry catalog (below): helm installs a chart, not a bundle.
scripts/build-catalog.sh <dir> builds the bundle from a checkout, for a fork or a local test: one catalog.tar.gz with the charts of argo-cd/apps and their dependencies vendored (so Argo CD needs no access to any upstream chart repository), the ohmlab chart and the example instance, from the files git tracks only. Push it as a single-layer OCI artifact (crane append --oci-empty-base -f catalog.tar.gz -t <registry>/<repo>:<version>) and declare that repository as a catalog.
The catalog released from this repository is oci://ghcr.io/this-is-tobi/ohmlab/catalog, one version per release of argo-cd/apps (OCI tag X.Y.Z, git tag catalog-vX.Y.Z). A release PR collects the conventional commits that touched argo-cd/apps, Renovate's chart bumps included (they are fix(deps), or feat(deps) for a major); merging it publishes the bundle, signs it, attaches a build provenance and then checks the published digest the way an adopter would (scripts/ci/verify-catalog.sh: anonymous pull, signature, provenance, every chart rendered from the bundle alone). Check a version before you pin it:

cosign verify ghcr.io/this-is-tobi/ohmlab/catalog:<version> \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/this-is-tobi/github-workflows/\.github/workflows/attest-docker\.yml@refs/tags/v[0-9]+(\.[0-9]+)*$'

Charts from a registry

By default every chart is a directory of the git repository. An entry can instead take a published Helm chart from an OCI registry: declare the registry once in instance.yaml, then point the entry at it.

yaml

# argo-cd/instances/<instance>/instance.yaml
catalogs:
  ohmlab: ghcr.io/this-is-tobi/charts # registry path, no oci:// scheme

# argo-cd/instances/<instance>/core.yaml
- app: ohmlab
  enabled: "true"
  catalog: ohmlab
  targetRevision: 0.1.3 # chart version, required
  releaseName: ohmlab
  namespace: argocd-system
  syncWave: -10

The chart name is the entry's chart (default: the app name) and the values still come from the instance values tree, so nothing else changes. A few rules keep this safe:

targetRevision is the exact chart version and has no default. An entry without it, or with a version range, is not pinned: leave a range out, since a registry tag can be re-pushed. The Application of an entry that omits it reports an invalid revision and no other app is affected.
A catalog name that is not declared in instance.yaml gives that Application no repository, so it is invalid instead of falling back to git.
The AppProject of the tier must list the registry path in sourceRepos (projects.<tier>.sourceRepos in the ohmlab values), spelled exactly like the catalog entry.
The manager chart reads catalogs from instance.yaml while the ApplicationSet reads the entries from core.yaml / tenant.yaml, on separate refreshes. An entry added in the same commit as its catalog can therefore show Unknown (invalid, nothing deployed or removed) for a few minutes until both have caught up. Declaring the catalog in an earlier change avoids it.
The ohmlab pin of an instance is bumped by Renovate (renovate.json watches the registry): merging its pull request is what updates the self-managed Argo CD.

Names

What belongs to an instance is named after it, so that a resource tells which instance it serves: its Vault KV mount, its Keycloak realm and the name that realm shows, and its Gateway (one name for every Gateway implementation, the parent of every route). The manager derives them from the instance's name, instance.name, which the root manager sets to the name of the instance's folder:

Name	Derived from `instance.name: mylab`	Used for
`vaultMount`	`mylab`	KV v2 mount of the tenant tier's secrets
`keycloakRealm`	`mylab`	Keycloak realm of the tenant apps
`keycloakRealmDisplayName`	`Mylab`	realm name on the Keycloak pages
`gateway`	`mylab-gateway`	Gateway, and the default parent of every route

What belongs to the ohmlab system keeps the ohmlab name whatever the instance, and the line follows the tiers. The core tier's secrets are in the system KV mount, ohmlab (systemMount in the vault-operator values); the tenant tier's are in the instance's mount. Each app's policy grants <system mount>/data/platforms/+/core/<app> and <instance mount>/data/platforms/+/tenant/<app>, the vault-operator chart creates both mounts, and the ohmlab controller's policy covers both. An instance whose core secrets are still in its own mount sets systemMount to that mount's name until they are copied. Keycloak follows the same line: the core apps sign in through the system realm, ohmlab (shown as Ohmlab), and the tenant apps through the instance's realm. The keycloak chart imports both (realms in its values, one file per realm, named after it); two realms with one name are one realm, which holds both tiers' clients.

These names are hard to change on a running cluster: a new mount starts empty, a renamed Gateway is recreated with its address and certificates, a new realm has no users. A cluster whose resources carry other names pins them in instance.yaml, each key on its own; the others stay derived:

yaml

# argo-cd/instances/<instance>/instance.yaml
names:
  vaultMount: secrets
  keycloakRealmDisplayName: My lab

Every Application of the instance gets the four names as Helm parameters, global.ohmlab.names.<key>. A chart takes its own explicit value first (for example gateway.gatewayName in an app's values), then the instance's name. A chart rendered without a manager falls back to the ohmlab names (ohmlab, Ohmlab, ohmlab-gateway).
The folder name has to be a lowercase DNS label of at most 55 characters, so that <name>-gateway is a valid Gateway name.
The manager chart refuses to render, instead of leaving a broken or derived name in place, on a key the product does not read, an empty name, a name with quotes, backslashes, braces or control characters, a Gateway that is not a DNS label, a realm or KV mount with characters Keycloak or Vault would not take, and a name those servers keep for themselves: the realm master (Keycloak's administration) and the mounts sys, auth, identity and cubbyhole (Vault's).
Every pull request of this repository renders the reference instance with the charts before and after the change, and fails on any object that would be deleted or renamed.

Instances set up before the names were derived ran on fixed names: the KV mount homelab for both tiers, the realm stored in Vault under the realm key of the Keycloak path, the display name Homelab, the Gateway homelab-gateway-traefik (Traefik) or homelab-gateway (APISIX, Cilium). Pin the ones your instance's name does not give before you upgrade, for example:

yaml

names:
  vaultMount: homelab
  keycloakRealm: homelab
  keycloakRealmDisplayName: Homelab
  gateway: homelab-gateway-traefik

Keep the core tier's secrets where they are too, with systemMount: homelab in values/core/vault-operator.yaml, until they are copied to ohmlab.

syncWave semantics: on an AppSet-generated Application, the sync-wave annotation alone orders nothing — apps sync in parallel and converge by retry. To actually enforce the ordering, enable progressive sync (progressiveSync.enabled: true in the instance-manager chart values); it maps waves onto ApplicationSet RollingSync steps. Requires the alpha ApplicationSetProgressiveSyncs feature gate, which the ohmlab chart enables on the core ArgoCD. It also turns automated sync off on the generated Applications: the ApplicationSet controller syncs an app again only when its revision changes, so drift at the same commit (a hand edit, a chart that renders more once a CRD exists) waits for the next commit or a manual sync.

Your network

The charts carry no network of their own. An instance sets its own ranges and addresses, and a chart that needs one stops rendering until it is set.

Value	App	Required	What it is
`lanOnly.sourceRange`	traefik	yes	The source ranges the `lan-only` Middleware lets through: your LAN or VPN, and the pod range of in-cluster callers of a gated UI.
`lanOnly.sourceRange`	longhorn, cilium	with `gateway.enabled`	Their own copy of that Middleware; keep the three in sync.
`cilium.k8sServiceHost`	cilium	yes	The address the Cilium agents reach the Kubernetes API at: an API server VIP or a control-plane node.
`crowdsecBouncer.clientTrustedIPs`	traefik	no, empty	Ranges that bypass the CrowdSec bouncer entirely, e.g. your LAN, so that a false positive cannot lock you out of it. A range here escapes every CrowdSec decision.
`crowdsec.config.postoverflows.s01-whitelist.whitelist-local.yaml`	crowdsec	no	The ranges CrowdSec never bans. The chart's file lists the cluster's pod and service ranges; to trust your LAN too, set the whole file with your LAN added.

The example instance uses documentation addresses (192.0.2.0/24, 192.0.2.10, RFC 5737), which match no real network: replace them.

What the apps expect from each other

Waves are not enforced by default (see the syncWave semantics above), so an app whose chart ships objects of another app's CRDs fails its first sync and converges once that app is up, provided it is up within the retry window: a failed automated sync is retried three times (about three and a half minutes) and then not again for the same commit. An app still failed after that stays so until you sync it once by hand or push a new commit. When you enable a subset, enable the providers first:

Needs	Provided by	Used by
Vault Secrets Operator CRDs (`secrets.hashicorp.com`)	`vault-operator`	Almost every chart: its secrets are `vso.vaultStaticSecrets` entries, which read the KV mount and path the instance's manager gives (`global.ohmlab.vault`)
Prometheus Operator CRDs (`monitoring.coreos.com`)	`prometheus-stack`	Charts that ship a ServiceMonitor, PodMonitor or PrometheusRule: switch their `monitoring` values off where you do not run it (`vault-operator` and `ohmlab-controller` leave theirs out by themselves when the CRDs are missing)
Gateway API CRDs	`./run.sh -b`	The routes of the apps exposed through a gateway

Vault, delivered by the Vault Secrets Operator, is the only secret backend supported today. Other backends (External Secrets, ...) are planned: every chart declares its secrets in one place (vso:) so they can be swapped later.

The _example values render for every chart (CI checks it) and every secret they declare is readable with the Vault policies and roles of the chart defaults (CI checks that too), so only the hostnames and similar are yours to set. The Vault KV mounts are ohmlab for the core tier and one named after the instance for the tenant tier (example there, see Names), both created by the vault-operator chart, whose policies follow them. The manager gives every app its tier's mount (global.ohmlab.vault.mount) and, with instance.env, the path prefix platforms/<instance.env>/<tier> (global.ohmlab.vault.pathPrefix), to which each chart adds its own segment (platforms/production/tenant/gitea): a secret sets its own mount or path only to read somewhere else. The example values run Argo CD with one replica of each component and no Redis HA, and Vault with one replica, so that they work on a single node; the chart defaults are highly available (three nodes or more), so drop those overrides there.

Secrets Management

Secrets live in Vault, delivered by the Vault Secrets Operator (see Secrets). During setup every password, token and so on that can be generated is randomly generated and stored in Vault; the credentials that cannot be generated (a GitHub App, a webhook URL, S3 keys) are declared as empty placeholders and set by hand. Nothing secret is committed.

Each vso.vaultStaticSecrets entry copies only what its consumer needs into its Kubernetes Secret: its destination.transformation sets excludeRaw: true and excludes: [".*"] and lists the keys to build under templates. Without that, the Vault Secrets Operator also writes the whole Vault path (as _raw) into the Secret, so a pod that mounts one Secret of an app would receive every credential of that app, such as the database superuser. CI refuses an entry that leaves them out (scripts/ci/check-vso-isolation.sh).

Optional: Sops-encrypted manifests

The sops app (sops-secrets-operator) and the ./run.sh -d / -e helpers are there for instances that prefer to commit encrypted Kubernetes manifests (a SopsSecret in the templates/ of a chart) next to their values. This repository's own instance does not use them. Prefer Vault where you can: the operator creates Secrets in any namespace, so it holds a cluster-wide permission on Secrets.

Have the ohmlab controller generate the age key pair in Vault, by adding this entry to ohmlab.vault.secrets in the vault-operator values (the controller must be enabled, see docs/08-controller.md), then enable the sops app in core.yaml (the example values of the app point at that path).
yaml
```
- path: ohmlab/platforms/production/core/sops
  data:
    secret: "<age:secret>"
    public: "<age:public>"
```

Create .sops.yaml at the root of the repository. The first age entry is the public key generated above, so the cluster can decrypt; add your own keys (age or PGP) so you can edit locally.

yaml

creation_rules:
- path_regex: \.yaml$
  encrypted_regex: ^(data|stringData)$
  key_groups:
  - age:
    - <public key of the cluster>
    - <your own public key>

Write the manifests as *.dec.yaml (git-ignored) under argo-cd/, encrypt them with ./run.sh -e and commit the *.enc.yaml files; ./run.sh -d decrypts them back for editing.

Deploy

Infrastructure

Deploy gateway and K3s cluster using the Ansible playbook:

# Update Ansible collections and deploy infrastructure
./run.sh -p ./ansible/install.yml -u -k

# Or with specific tags
./run.sh -p ./ansible/install.yml -t gateway   # Deploy gateway only
./run.sh -p ./ansible/install.yml -t k3s       # Deploy K3s cluster only

The -k flag fetches the kubeconfig from the master node and merges it into your local kubeconfig.

Applications (GitOps)

Once the infrastructure is ready, bootstrap the GitOps stack with a single command:

# Set kubectl context
kubectl config use-context homelab

# Bootstrap (or upgrade) the mylab instance
./run.sh -b mylab

This installs the ohmlab Helm release in the argocd-system namespace, which contains:

The core ArgoCD instance (engine; not user-facing).
The root manager ApplicationSet (discovers every instance under argo-cd/instances/*).
The admin-core and admin-tenant AppProjects.

The chart is the local ./charts/ohmlab, unless the ohmlab entry of the instance's core.yaml has a catalog (see Charts from a registry): then the published chart at the exact version pinned there is installed, which is the chart the self-managed Application runs afterwards. A catalog that is not declared in instance.yaml, or a version that is not exact, stops the script before it touches the cluster.

The root manager then renders one instance-<name> Application per discovered folder. That Application points at the ./argo-cd/apps/instance-manager chart, which produces two child ApplicationSets (core-<name> and tenant-<name>). The first sync wave (-10) reconciles ohmlab itself onto its chart — from git by default, or from a pinned registry version with catalog (see Charts from a registry) — and the bootstrap release is then self-managed.

Notes:
Multiple tags can be passed as follows: ./run.sh -p ./ansible/install.yml -t gateway,k3s
First gateway init can take a long time to run because of OpenVPN key generation (5-10min).
Bootstrap admin password: pass ARGOCD_ADMIN_PASSWORD=mypass ./run.sh -b mylab to set it explicitly. Without this var, the ArgoCD chart auto-generates a password and stores it in argocd-initial-admin-secret; the script prints it at the end of the run.
OIDC for the core ArgoCD is intentionally disabled at bootstrap. Enable it in argo-cd/instances/<instance>/values/core/ohmlab.yaml once Keycloak is ready (uncomment the oidc.config block and provide the client secret out-of-band).

Using an existing cluster

The GitOps layer (argo-cd/) is independent of how the cluster was created — it only needs a kubeconfig with cluster-admin. On managed Kubernetes (EKS/GKE/AKS/...) or any pre-existing cluster, skip the Ansible phase entirely and run ./run.sh -b <instance> against your context.

Create your own instance folder — copy argo-cd/instances/_example and see Adding a new instance. Points worth checking for a non-K3s cluster:

Storage — longhorn assumes bare-metal disks. On a cloud provider, disable it and use the provider's own StorageClass (set it as default, or set storageClass explicitly on the apps that request one).
Ingress exposure — the traefik app's Service is type LoadBalancer. A cloud provider assigns a real external IP automatically, so drop the spec.externalIPs list this repo uses (that exists only because it runs bare metal without an LB controller).
CNI — managed clusters ship their own; leave the cilium app disabled. Read the CNI caveat at the top of any templates/networkpolicy.yaml before enabling networkPolicy — the rules were verified against Cilium and rely on three CNI-specific behaviours.
Pod/service CIDRs — anything referencing 10.42.0.0/16 / 10.45.0.0/16 / 10.43.0.0/16 (NetworkPolicy ipBlock rules, crowdsec's allowed_ranges and its whitelist) must match your cluster's actual ranges, and so must the pod range in your lanOnly.sourceRange (see Your network).
Node labels — apps pinning workloads with nodeSelector: {node-type: worker} (CNPG databases, Vault) need that label to exist, or the selector removed.
Gateway-host services — HAProxy, PiHole, WireGuard and the gateway CrowdSec engine are Ansible-managed Docker services on a separate host. They have no in-cluster equivalent; ignore them unless you replicate that setup.

Destroy

It is possible to cleanly destroy the K3s cluster by running:

# Destroy cluster
./run.sh -p ./ansible/install.yml -t k3s-destroy

Maintenance

OS upgrades

Run an OS package upgrade on all managed hosts (gateway + K3s nodes), rebooting only if required:

./run.sh -p ./ansible/install.yml -t os-upgrade

Debian major version upgrade

Upgrade all hosts in-place from one Debian release to the next (e.g. bookworm → trixie). K3s nodes are automatically drained before the upgrade and uncordoned after reboot. Hosts are processed one at a time (serial: 1).

Set the target release in inventory/group_vars/all.yml (or pass it as extra var):

yaml

common_dist_upgrade_target_release: trixie

Run the dist-upgrade:

./run.sh -p ./ansible/install.yml -t dist-upgrade

Notes:
Hosts already running the target release are automatically skipped.
Ensure kubectl is configured locally — the drain/uncordon commands run from your workstation.
After a major upgrade, re-run the full infra playbook to reconcile Docker repos and other codename-dependent configuration: ./run.sh -p ./ansible/install.yml -u

Ansible collection updates

Update pinned Ansible Galaxy collections to the latest compatible version before running a playbook:

./run.sh -p ./ansible/install.yml -u

Collection version ranges are pinned in ansible/collections/requirements.yml. Bump the major range when upgrading to a new major release.

Ansible vault

Edit encrypted secrets:

cd ansible && ansible-vault edit inventory/group_vars/vault.yml

Docker image updates (gateway)

Image versions for gateway services (HAProxy, PiHole, WireGuard) are managed in inventory/group_vars/gateway.yml. Update the version field and re-run the gateway playbook:

./run.sh -p ./ansible/install.yml -t gateway

K3s version updates

The K3s version is pinned in two places that Renovate bumps together in one PR:

system-upgrade.k3s.version in argo-cd/instances/<instance>/values/core/system-upgrade.yaml: the version the running cluster is upgraded to. The system-upgrade-controller applies it in-cluster, control-plane nodes one at a time then the agents, inside the maintenance window set in argo-cd/apps/system-upgrade/values.yaml (every night, 02:00–06:00 Europe/Paris).
k3sVersion in inventory/group_vars/k3s.yml (plus the example inventory and the role default): the version ansible installs on new or re-provisioned nodes.

Merging the PR is the upgrade: it starts in the next window, or right away if merged during one. Follow it with kubectl -n system-upgrade get plans,jobs and kubectl get nodes.

Kubernetes does not support skipping a minor version, so Renovate opens one PR per minor, and minor bumps wait for approval on the dependency dashboard (patch bumps open on their own, after a week of soak). Before approving a minor:

Check that every chart deployed on the cluster supports the new Kubernetes minor.
In the same PR, bump by hand KUBECTL_VERSION in cli/Dockerfile and controller.job.kubectlImage in argo-cd/apps/system-upgrade/values.yaml to that minor (Renovate only moves their patch version).

Kubernetes application updates

Application chart versions are managed via GitOps — update the Helm chart version in the relevant argo-cd/apps/<app>/Chart.yaml and push. ArgoCD auto-syncs the change.

Releases

Three parts are released from this repository, each with its own version, release PR, changelog, git tag and GitHub release:

Part	Published as	Git tag	Changelog
`ohmlab` CLI (`cli/`)	binaries on the GitHub release, image `ghcr.io/this-is-tobi/ohmlab/cli:X.Y.Z`	`vX.Y.Z`	`cli/CHANGELOG.md`
bootstrap chart (`charts/ohmlab`)	`oci://ghcr.io/this-is-tobi/charts/ohmlab`, version `X.Y.Z`	`chart-vX.Y.Z`	`charts/ohmlab/CHANGELOG.md`
catalog (`argo-cd/apps`)	`oci://ghcr.io/this-is-tobi/ohmlab/catalog:X.Y.Z`	`catalog-vX.Y.Z`	`argo-cd/apps/CHANGELOG.md`

release-please opens one release PR per part from the conventional commits under its directory (Renovate's dependency bumps there are fix(deps), feat(deps) for a major); merging it publishes that part only, signed and with a build provenance, and the chart and the catalog are then checked the way an adopter would use them (anonymous pull of the published digest, signature, provenance, render with the example instance: scripts/ci/verify-chart.sh, scripts/ci/verify-catalog.sh). The CLI release stays marked "Latest". The chart README is generated by helm-docs from charts/ohmlab/README.md.gotmpl (helm-docs --chart-search-root charts/ohmlab); CI fails when it is stale.

Architecture

Two ArgoCD instances

The cluster runs two ArgoCD instances with very different roles:

Instance	Namespace	Purpose
core	`argocd-system`	The engine. Runs the `manager` ApplicationSet that drives every other app. Not user-facing.
personal	`argo-cd`	A user-facing sandbox at `gitops.<domain>`. Driven by core, but has no `manager` itself — used through the UI for ad-hoc deployments.

Both instances are deployed from the same chart at argo-cd/apps/argo-cd/, differentiated by their per-instance values files.

App-of-apps flow

Adding a new instance is purely declarative — just create a folder under argo-cd/instances/ (with instance.yaml + core.yaml + tenant.yaml) and a matching argo-cd/instances/<name>/values/{core,tenant}/ tree. The root manager picks it up on its next reconciliation:

Create argo-cd/instances/<name>/instance.yaml (cluster destination, repos, project bindings).
Create argo-cd/instances/<name>/core.yaml and/or argo-cd/instances/<name>/tenant.yaml.
Create argo-cd/instances/<name>/values/core/ and/or argo-cd/instances/<name>/values/tenant/ with at least a ohmlab.yaml (under core/) for the self-managed bootstrap App when shipping core on that cluster.
Bootstrap the first instance with ./run.sh -b <name> against its target cluster; subsequent instances are then picked up automatically by the existing root manager.

Topologies

The two-level pattern accommodates very different deployment models, all driven by the same root manager and chart catalog:

Topology	Folders on disk	Where things run
All-in-one	`mylab/` with both `core.yaml` and `tenant.yaml`	Single cluster, single ArgoCD, both AppSets land on `in-cluster`.
SaaS shared core	`saas-admin/` (core only) + N × `saas-customer-*/` (tenant only)	Admin cluster runs the core stack; each customer gets its own cluster for tenants.
Dedicated core	One pair per org: `<org>-admin/` (core) + `<org>-prod/` (tenant)	Strict per-org isolation: dedicated admin cluster + dedicated app cluster.

The target cluster for each instance is set in its instance.yaml.destination.server; remote clusters are registered in ArgoCD via Cluster secrets (managed via Vault/VSO if desired).

Tier-flexible apps

A handful of apps don't naturally belong in a single tier — they're needed wherever workloads run, regardless of whether the cluster is acting as a "core/admin" cluster or a "tenant/apps" cluster. The catalog lists them in both _example/core.yaml and _example/tenant.yaml with appropriate per-tier syncWaves; for any concrete instance you enable the entry in exactly one tier and leave the other disabled.

App	Where to enable
`cert-manager`	Wherever Ingress / TLS certificates are issued.
`traefik`	Wherever an ingress controller is needed.
`keycloak`	Wherever the SSO IdP runs (often tenant in SaaS, core in all-in-one).
`kubernetes-dashboard`	Wherever cluster admins want a UI; one per cluster.
`longhorn`	Wherever block storage is needed (typically every cluster with stateful workloads).
`prometheus-stack`	Wherever observability is collected (often core in shared topologies).
`teleport`	Wherever the access proxy runs.
`vault`	Wherever the secrets backend lives (often co-located with workloads consuming it).

Enabling a tier-flexible app in both tiers of the same instance would create two Applications with the same name and is not supported. The _example template ships with everything disabled to make this an explicit, deliberate choice.

Splitting infra and values across two repositories

Per-instance values live alongside the rest of the instance metadata (argo-cd/instances/<inst>/values/{core,tenant}/<app>.yaml). For most setups keeping everything in a single repo is the simplest option.

If you need to keep the chart catalog public while keeping per-instance values (which often contain hostnames, secrets references, OIDC client IDs, etc.) in a private repository, the instance-manager chart supports it natively:

yaml

# argo-cd/instances/<inst>/instance.yaml
repoURL: https://github.com/this-is-tobi/ohmlab.git # public: charts + this file
valuesRepoURL: https://github.com/<you>/mylab-private.git # private: values tree only
targetRevision: main

When valuesRepoURL is set, the child AppSets attach two git sources to every generated Application:

repoURL (the chart source) — used to fetch argo-cd/apps/<app>/.
valuesRepoURL (the $values ref) — used to fetch argo-cd/instances/<inst>/values/<tier>/<app>.yaml.

In the values repository, mirror the path layout exactly:

mylab-private/                                # your private repo root
└─ argo-cd/
   └─ instances/
      └─ <inst>/
         └─ values/
            ├─ core/<app>.yaml
            └─ tenant/<app>.yaml

Per-app overrides also exist if a single app needs a different layout — valuesPath overrides the whole path, useful for charts that ship in their own repository entirely.

Keeping the instance in a repository of its own

The product (the charts and the instance-manager chart) and an instance (its instance.yaml, core.yaml, tenant.yaml and values) can live in two repositories, and the product can come from git or from the published OCI catalog. The root manager reads the instances from one repository and its chart from the other:

yaml

# argo-cd/instances/<inst>/values/core/ohmlab.yaml (the bootstrap chart)
rootManager:
  repoURL:
    sources: https://github.com/this-is-tobi/ohmlab.git # the instance-manager chart: a git URL, or oci://<registry>/<path>/catalog
    values: https://github.com/<you>/<instance-repo>.git  # where argo-cd/instances/* lives
  sourcesRevision: ""  # revision of `sources` alone: a git revision, or the exact version of an oci:// bundle
  targetRevision: main # revision of the values repository

The instance then takes its tier files and values from its own repository and its charts from a catalog:

yaml

# argo-cd/instances/<inst>/instance.yaml
repoURL: https://github.com/<you>/<instance-repo>.git # this file, core.yaml, tenant.yaml and the values
catalogs:
  product:
    repoURL: https://github.com/this-is-tobi/ohmlab.git # or oci://<registry>/<path>/catalog
    version: main                                        # a git revision, or the exact bundle version
defaultCatalog: product

With an oci:// source, rootManager.repoURL.values and an exact rootManager.sourcesRevision are required: a bundle cannot be listed for instances, and an unset version would follow the newest one. The render fails without them.
When projects.<scope>.sourceRepos is empty, an AppProject allows the sources and values repositories plus projects.<scope>.extraSourceRepos (a registry catalog, another bundle). A bundle is listed with its oci:// scheme, a registry path without one. An explicit list is kept as written.
Bootstrap from a checkout of the product, pointing at the instance folders: ./run.sh -b <inst> -i ../<instance-repo>/argo-cd/instances.
The Ansible inventory can move with the instance. Point ANSIBLE_INVENTORY at a directory laid out like ansible/inventory-example: ANSIBLE_INVENTORY=../<instance-repo>/ansible/inventory ./run.sh -p ./ansible/install.yml. The files the roles generate (.k3sToken, .k3sCaData, .pihole_password, .wireguard_password, and .pi_password from scripts/setup-pi.sh) are written next to that inventory, so the instance repository has to ignore them. Set ANSIBLE_VAULT_PASSWORD_FILE when the vault password is not ansible/.vault_password, and give k3sCniChartValues an absolute path, such as "{{ inventory_dir }}/../../argo-cd/instances/<inst>/values/core/cilium.yaml".
An instance repository runs the product's checks on its pull requests by calling .github/workflows/validate-instance.yml of the product, with ref (the product revision it consumes), product-repository and instances-path. The checks are that every top-level values key reaches a chart (Helm drops one that names no dependency and is read by no template, without a word), Vault access, the VaultSeed objects, VSO isolation, and a render of every app the instance enables with its values (scripts/ci/render-instance.sh). The product can call the same workflow on an instance repository (instance-repository, instance-ref) to check its own changes against that instance before they merge. With base-ref (a pull request's base), it renders the instance from both revisions and fails when the change would delete or rename an object on that instance's cluster; objects whose content changes are listed in the job summary, which is where a renamed Vault mount or Keycloak realm shows.

Sync waves

Apps are reconciled in syncWave order. The ordering of the reference deployment:

Wave	Tier	Apps
-10	core	`ohmlab` (self)
0	core	`longhorn`
10	core	`cert-manager`, `vault-operator`
11	core	`traefik`
13	core	`trust-manager`
15	core	`kyverno`
20	core	`cloudnative-pg`
50	core	`prometheus-stack`
55	core	`keycloak`
60	core	`crowdsec`, `system-upgrade`, `teleport`
100	tenant	`argo-cd` (personal)
110	tenant	`gitea`, `mattermost`, `rustfs`
200	tenant	`homepage`

Core Services

Core services provide the foundation for the platform:

Longhorn - storage management in the cluster.
Traefik - ingress controller & Gateway API implementation to expose services.
Cert-Manager - certificate management for TLS.
Vault Operator - secret management for services deployments.
Trust-Manager - publishes Vault's CA as a ConfigMap for Traefik's backend TLS validation.
Kyverno - admission policy enforcement.
ArgoCD - deployment management following GitOps.
CloudNative-PG - PostgreSQL operator for databases.

Platform Services

Platform services are deployed on top of core services:

Keycloak - identity and access management (SSO).
Gitea - self-hosted Git service.
Mattermost - team communication.
*And more...

Known issues

At the moment, mattermost and outline images are not arm64 compatible so their deployment are using custom mirror image with compatibility (see. this repo and associated ArgoCD applications).

The official Harbor helm chart cannot be used due to arm64 incompatibility, the Bitnami distribution is used instead.

Installation ​

Prerequisites ​

Ansible Vault ​

Settings ​

Infrastructure ​

Applications (GitOps) ​

Catalogs ​

Charts from a registry ​

Names ​

Your network ​

What the apps expect from each other ​

Secrets Management ​

Optional: Sops-encrypted manifests ​

Deploy ​

Infrastructure ​

Applications (GitOps) ​

Using an existing cluster ​

Destroy ​

Maintenance ​

OS upgrades ​

Debian major version upgrade ​

Ansible collection updates ​

Ansible vault ​

Docker image updates (gateway) ​

K3s version updates ​

Kubernetes application updates ​

Releases ​

Architecture ​

Two ArgoCD instances ​

App-of-apps flow ​

Topologies ​

Tier-flexible apps ​

Splitting infra and values across two repositories ​

Keeping the instance in a repository of its own ​

Sync waves ​

Core Services ​

Platform Services ​

Known issues ​

Installation

Prerequisites

Ansible Vault

Settings

Infrastructure

Applications (GitOps)

Catalogs

Charts from a registry

Names

Your network

What the apps expect from each other

Secrets Management

Optional: Sops-encrypted manifests

Deploy

Infrastructure

Applications (GitOps)

Using an existing cluster

Destroy

Maintenance

OS upgrades

Debian major version upgrade

Ansible collection updates

Ansible vault

Docker image updates (gateway)

K3s version updates

Kubernetes application updates

Releases

Architecture

Two ArgoCD instances

App-of-apps flow

Topologies

Tier-flexible apps

Splitting infra and values across two repositories

Keeping the instance in a repository of its own

Sync waves

Core Services

Platform Services

Known issues