Self-hosting

Lucity is two Helm charts on a Kubernetes cluster. lucity-infra brings the pieces the platform leans on, and lucity brings the platform itself. Nothing phones home, nothing is licensed, and the cluster stays yours.

That said, this is not a one-command install. The platform runs other people's code on shared infrastructure, so it wants TLS, DNS, an identity provider, and a container registry before it will do anything useful. Budget an afternoon for the first one.

Prerequisites

A cluster, first of all:

And two domains, which must be different from each other:

Workload subdomains are handed to arbitrary user code, so anything sharing that domain shares cookies and certificate scope with it. Our own split is lucity.cloud for the platform and lucity.app for workloads.

You will also need a GitHub App, which is how the platform reads source repositories, and an object store for buckets.

What is required and what is not

The infra chart ships a lot of switches. Only some of them are load-bearing:

Before the charts

Two things have to be true about the cluster before either chart will behave.

Let containerd pull from the in-cluster registry

The platform pulls user images from Zot over plain HTTP on a fixed service IP, and containerd will not do that by default. The infra chart runs a DaemonSet that writes the per-registry hosts.toml to every node, but containerd only reads it if its config points there:

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"

Containerd loads one config file rather than merging them, so add that block to a full copy of your nodes' existing config and restart containerd. On some distributions, Flatcar among them, the running config lives outside /etc and the service has to be pointed at the new file explicitly.

Give cert-manager DNS-01 access

Platform certificates include a wildcard for the workload domain, and a wildcard can only be validated over DNS-01. So cert-manager needs an API credential for your DNS provider that can write TXT records in both zones. Scope it to those zones and nothing else: it is the one credential here that can change public DNS.

How you create it depends on the provider, and cert-manager documents a solver for each of the common ones(opens in a new tab). Whichever you use, the shape is the same. Put the provider token in a Secret in the cert-manager namespace:

kubectl create secret generic dns-credentials \
  -n cert-manager \
  --from-literal=api-token='<provider-api-token>'

Then reference it from an issuer in your values file, rather than applying the issuer by hand, with one solver per zone selected by dnsZones:

clusterIssuers:
  - name: letsencrypt-dns01
    acme:
      server: https://acme-v02.api.letsencrypt.org/directory
      email: you@example.com
      solvers:
        - selector:
            dnsZones: ["<platform-domain>"]
          dns01:
            # provider-specific block, see the cert-manager docs
        - selector:
            dnsZones: ["<workload-domain>"]
          dns01:
            # same provider, the other zone

If the two domains live with different providers, give each solver its own provider block.

A second issuer, using HTTP-01, handles certificates for the custom domains your users attach to their own services. That one needs no credentials, because it validates over the gateway that is already serving the domain.

The infra chart

Two files drive this chart. infra-values.yaml holds the shape of your installation:

zot:
  persistence: true
  pvc:
    create: true
    storage: 20Gi
    storageClassName: <storage-class>
  service:
    # Pin this inside your service CIDR. Nodes pull images by this address,
    # so it has to be stable and it has to match registryPull.host later.
    clusterIP: "10.96.100.100"
  mountConfig: true

gateway:
  className: cilium
  listeners:
    - name: platform-http
      protocol: HTTP
      port: 80
      hostname: <platform-domain>
    - name: platform-https
      protocol: HTTPS
      port: 443
      hostname: <platform-domain>
      tls:
        mode: Terminate
        certificateRefs:
          - name: platform-tls
    - name: workload-http
      protocol: HTTP
      port: 80
      hostname: "*.<workload-domain>"
    - name: workload-https
      protocol: HTTPS
      port: 443
      hostname: "*.<workload-domain>"
      tls:
        mode: Terminate
        certificateRefs:
          - name: workload-tls
    - name: id-http
      protocol: HTTP
      port: 80
      hostname: id.<platform-domain>
    - name: id-https
      protocol: HTTPS
      port: 443
      hostname: id.<platform-domain>
      tls:
        mode: Terminate
        certificateRefs:
          - name: id-tls

certificates:
  - name: platform-tls
    secretName: platform-tls
    issuerRef: { name: letsencrypt-dns01 }
    dnsNames: ["<platform-domain>"]
  - name: workload-tls
    secretName: workload-tls
    issuerRef: { name: letsencrypt-dns01 }
    dnsNames: ["*.<workload-domain>"]
  - name: id-tls
    secretName: id-tls
    issuerRef: { name: letsencrypt-dns01 }
    dnsNames: ["id.<platform-domain>"]

clusterIssuers: # both issuers from the previous section

logto:
  enabled: true
  endpoint: https://id.<platform-domain>
  postgres:
    storageClass: <storage-class>

victoria-metrics-single:
  enabled: true
  server:
    persistentVolume:
      storageClassName: <storage-class>

victoria-logs-single:
  enabled: true
  server:
    persistentVolume:
      storageClassName: <storage-class>

otelCollector:
  daemonset: { enabled: true }
  deployment: { enabled: true }

Every listener needs a matching certificate, and the certificate names here are what the listeners reference.

infra-secrets.yaml holds the credentials. The registry needs two users, one that pushes built images and one that nodes pull with:

htpasswd -bBn builder "$(openssl rand -hex 16)"  > htpasswd
htpasswd -bBn reader  "$(openssl rand -hex 16)" >> htpasswd

Keep both plaintext passwords, because the platform chart needs them again. Then:

zot:
  configFiles:
    htpasswd: |
      # both lines from the file above

logto:
  secrets:
    secretVaultKek: "<openssl rand -base64 32>"

Install lucity-infra in two passes the first time. The chart carries both the CloudNativePG operator and a PostgreSQL cluster that needs its CRDs, so a single-pass install cannot resolve them.

Pass one brings up the operator with the identity provider switched off:

helm upgrade --install lucity-infra oci://ghcr.io/zeitlos/lucity/charts/lucity-infra \
  --version <chart-version> -n lucity-system --create-namespace \
  -f infra-values.yaml -f infra-secrets.yaml --set logto.enabled=false

Wait for the operator and its webhook:

kubectl -n lucity-system wait --for=condition=Available deploy/lucity-infra-cloudnative-pg --timeout=180s

Then pass two, with nothing overridden:

helm upgrade --install lucity-infra oci://ghcr.io/zeitlos/lucity/charts/lucity-infra \
  --version <chart-version> -n lucity-system \
  -f infra-values.yaml -f infra-secrets.yaml

Upgrades after this are single-pass, because the CRDs are already in the cluster.

Give the identity provider a minute to pick up its database credentials, then check that everything is up:

kubectl -n lucity-system get pods
kubectl -n lucity-system get cluster
kubectl -n lucity-system get certificate
kubectl get gateway -A

The gateway should report an address and PROGRAMMED: True.

DNS for the platform

The gateway now has an address, so the platform hostnames can point at it. Three records carry the whole installation:

The wildcard is what makes a freshly generated service hostname work immediately, with no DNS wait and no per-service record.

Once a wildcard covers the workload domain, the only records left are those two platform hostnames, which never change, so external-dns buys you little here.

Bootstrapping the identity provider

Logto starts empty and the platform expects a specific shape inside it. Nothing automates this yet, so it is console work.

The console is not exposed publicly, so reach it over a port-forward:

kubectl -n lucity-system port-forward svc/lucity-infra-logto 3002:3002

Open http://localhost:3002 and create the first admin account. Then work through the sections below in order.

Workspaces are Logto organizations, and a workspace member's rights come from their organization role. Build this in the order below, because the permissions live on the API resource rather than on the organization.

API resource

Create the API resource first. Its identifier is the audience you will give the conductor, for example https://api.<your-platform-domain>, and it becomes the OIDC_AUDIENCE value.

Then give that resource three permissions: admin, member, and deployer.

Organization roles

Now create the roles under Organization template. Each asks for a name, a description, and a role type. Assign the API resource permissions from the previous step, leaving organization permissions empty:

The two User roles cover people signing in. The three machine-to-machine roles cover workspace API keys and CI, so machine-deployer is what a GitHub Actions deploy runs as.

Applications

Three, because three different clients sign in:

The CLI logs in with PKCE against a loopback listener and takes the first free port, so register all three.

Give the M2M application the Logto Management API role. The conductor uses it to create organizations, assign roles, and read members, so without it workspace creation fails.

GitHub connector

Users sign in to Logto, Logto runs the GitHub handshake, and the resulting token is kept in Logto's Secret Vault, so GitHub redirects back to Logto and not to the platform.

Add a GitHub social connector. Three fields on that page matter.

Identity provider name must be github, which is also the default.

Client ID and Client Secret, under Parameter configuration, take the values from your GitHub App, not from a separate OAuth App. Leave Scope empty: a GitHub App's access is decided by where it is installed, and it ignores OAuth scopes entirely.

Store tokens for persistent API access in General settings has to be turned on, because it is off by default and the platform reads the stored GitHub token from the Secret Vault.

The same page shows the Redirect URI (Callback URI) that GitHub must send users back to:

https://<logto-endpoint>/callback/<connector-id>

The connector ID is generated when the connector is created, so this cannot be known in advance. Copy it and add it to your GitHub App's callback URL list. A GitHub App accepts several, so adding it after the fact disturbs nothing.

Under Sign-in and account → Sign-up and sign-in, add GitHub under Social sign-in and leave the identifier lists empty, so "Continue with GitHub" is the only way in. Leave Enable user registration on unless you want an invite-only installation.

Account API

On the Account center tab of that same page, switch the account API on. Every self-service field underneath it can stay Off.

The GitHub App

The App is what lets the platform read source repositories and hear about pushes. Create it under the account or organization that owns the repositories you want to deploy, and give it these settings in full.

URLs

Three of them, and they point in two different directions, which is the part people get wrong:

The callback goes to the identity provider, not to the platform, because the identity provider runs the GitHub sign-in handshake. The other two go to the platform. You get the connector id when you create the GitHub connector, so fill the callback in afterwards.

Webhook

Subscribe to exactly one event: Push. That is what triggers an automatic deploy on a branch the platform is watching. Nothing else is read, so leave the rest unsubscribed.

Permissions

The platform only ever reads your code, so every permission it needs is read-only:

Private key

At the bottom of the App's General page, click Generate a private key. GitHub downloads a .pem and shows it to you once, so keep it somewhere safe or generate a fresh one later. Its contents go into githubPrivateKey in the platform secrets, and the conductor reads it back from the path in GITHUB_PRIVATE_KEY_PATH.

Note the App ID, the Client ID, and a generated client secret from the same page while you are there. The client secret is also what the identity provider's GitHub connector needs.

Finally, install the App on the account or organization holding the repositories you want to deploy.

The platform chart

With the identity provider configured, the platform chart has everything it needs. Its values file is where the identifiers you collected come together:

Which comes together as values.yaml:

registryPull:
  host: "10.96.100.100:5000"   # the Zot clusterIP you pinned
  username: reader

routes:
  enabled: true
  hostname: <platform-domain>

conductor:
  registryHost: lucity-infra-zot.lucity-system.svc.cluster.local:5000
  registryUsername: builder
  buildkit:
    enabled: true
    storageSize: 20Gi
  env:
    OIDC_ISSUER_URL: https://id.<platform-domain>/oidc
    OIDC_AUDIENCE: https://api.<platform-domain>
    OIDC_CLIENT_ID: <dashboard-app-id>
    OIDC_CLI_CLIENT_ID: <cli-app-id>
    OIDC_CALLBACK_URL: https://<platform-domain>/auth/callback
    DASHBOARD_URL: https://<platform-domain>/app

    GITHUB_APP_ID: "<app-id>"
    GITHUB_APP_SLUG: <app-slug>
    GITHUB_CLIENT_ID: <client-id>
    GITHUB_PRIVATE_KEY_PATH: /secrets/github/github-app.pem

    LOGTO_ENDPOINT: http://lucity-infra-logto.lucity-system.svc.cluster.local:3001
    VICTORIA_METRICS_URL: http://lucity-infra-victoria-metrics-single-server.lucity-system.svc.cluster.local:8428

    INTERNAL_JWT_PRIVATE_KEY_PATH: /secrets/internal-jwt/private.pem
    INTERNAL_JWT_PUBLIC_KEY_PATH: /secrets/internal-jwt/public.pem

    REGISTRY_URL: lucity-infra-zot.lucity-system.svc.cluster.local:5000
    REGISTRY_PUSH_URL: lucity-infra-zot.lucity-system.svc.cluster.local:5000
    REGISTRY_PULL_URL: 10.96.100.100:5000
    REGISTRY_AUTH_SECRET: lucity-registry-auth

    WORKLOAD_DOMAIN: <workload-domain>
    DATABASE_DOMAIN: db.<workload-domain>
    IP_ADDRESS: "<gateway-address>"

    OVH_PROJECT_ID: "<ovh-project-id>"
    OVH_ENDPOINT: ovh-eu
    OVH_REGION: GRA
  envSecret:
    SESSION_SECRET: SESSION_SECRET
    GITHUB_CLIENT_SECRET: GITHUB_CLIENT_SECRET
    WEBHOOK_SECRET: GITHUB_WEBHOOK_SECRET
    LOGTO_M2M_APP_ID: LOGTO_M2M_APP_ID
    LOGTO_M2M_APP_SECRET: LOGTO_M2M_APP_SECRET
    OVH_APPLICATION_KEY: OVH_APPLICATION_KEY
    OVH_APPLICATION_SECRET: OVH_APPLICATION_SECRET
    OVH_CONSUMER_KEY: OVH_CONSUMER_KEY

cashier:
  enabled: false

envSecret maps an environment variable to a key in secrets.yaml, which holds everything sensitive:

secrets:
  SESSION_SECRET: "<openssl rand -hex 32>"
  GITHUB_WEBHOOK_SECRET: "<same value as the GitHub App webhook secret>"
  GITHUB_CLIENT_SECRET: "<from the GitHub App>"
  REGISTRY_PASSWORD: "<the builder password>"
  LOGTO_M2M_APP_ID: "<the M2M application's App ID>"
  LOGTO_M2M_APP_SECRET: "<the M2M application's App Secret>"
  OVH_APPLICATION_KEY: ""
  OVH_APPLICATION_SECRET: ""
  OVH_CONSUMER_KEY: ""

githubPrivateKey: |
  -----BEGIN RSA PRIVATE KEY-----
  ...the .pem you downloaded from the GitHub App
  -----END RSA PRIVATE KEY-----

registryPull:
  password: "<the reader password>"

internalJWT:
  privateKey: |
    -----BEGIN EC PRIVATE KEY-----
  publicKey: |
    -----BEGIN PUBLIC KEY-----

The internal keypair signs the tokens the control plane issues to its own build and deploy jobs. Generate it with:

openssl ecparam -name prime256v1 -genkey -noout -out internal-jwt-private.pem
openssl ec -in internal-jwt-private.pem -pubout -out internal-jwt-public.pem

Object storage credentials are also required, and the conductor will not start without them. Today the only implemented backend is OVH, so you need OVH_APPLICATION_KEY, OVH_APPLICATION_SECRET, OVH_CONSUMER_KEY and OVH_PROJECT_ID. Scope them to the one project the platform should use, because it creates and deletes buckets and credentials there on its own.

Then install, the same way as the infra chart and with the same caution about versions:

helm upgrade --install lucity oci://ghcr.io/zeitlos/lucity/charts/lucity \
  --version <chart-version> -n lucity-system \
  -f values.yaml -f secrets.yaml

A healthy conductor says so plainly in its logs, and the line worth looking for is the one proving it reached the identity provider and found your roles:

INFO logto org roles cached admin=... member=...

Verifying the install

A few checks, fastest first:

kubectl -n lucity-system get pods
kubectl -n lucity-system get certificate
kubectl -n lucity-system get gateway lucity-gateway \
  -o jsonpath='{range .status.listeners[*]}{.name}{": "}{range .conditions[*]}{.type}={.status} {end}{"\n"}{end}'

Every certificate True, and every listener Programmed=True. A listener stuck unprogrammed is nearly always waiting on its certificate.

Then the platform itself, which should answer on each path the route table sends somewhere different:

curl -o /dev/null -w '%{http_code}\n' https://<platform-domain>/
curl -o /dev/null -w '%{http_code}\n' https://<platform-domain>/app/
curl https://<platform-domain>/auth/config

/auth/config is the most useful of the three, because it echoes back the issuer, audience, and CLI client ID the conductor actually loaded.

Finally, confirm nodes can pull from the registry by asking for an image that does not exist:

kubectl run pulltest --image=<registry-service-ip>:5000/does-not-exist:test --restart=Never --command -- true
kubectl describe pod pulltest

An auth or not-found error is success, because containerd reached the registry and got an answer. Delete the pod afterwards.

Sign in at https://<platform-domain>/app/, and the first account to authenticate gets its own workspace.

Reference profiles

The repository carries the complete values files for the clusters we run, under deployments/. They are the fastest way to see how the pieces fit together, and the closest thing to a known-good starting point:

• deployments/lucity-prod/ is the production profile, with billing, backups, analytics, alerting, and database exposure all switched on
• deployments/lucity-dev/ is a smaller three-node profile with those switched off, which is roughly the minimum that still runs real workloads
• deployments/minikube/ is the local development profile

Copy whichever is closest, then work through the values that name a domain, a storage class, or a credential.