Kubernetes

The chart lives in pnex-deploy (helm/pnex). It deploys the PNeX server and job worker, Rauthy, OpenObserve, Valkey, RustFS and a PostgreSQL cluster.

This page walks through a real install of 0.1.0-beta.2, recorded on a fresh Ubuntu VM with a public IP. The demo uses k3s because it sets up a single-node cluster in one command, with an ingress controller (Traefik) and a default StorageClass included. Any conformant Kubernetes works: managed (EKS, GKE, AKS, Scaleway Kapsule…), RKE2, microk8s, kubeadm… Only the ingress class changes.

RequirementsCopy Anchor Link

• Kubernetes 1.27 or later, a default StorageClass.
• An ingress controller (Traefik in the demo, ingress-nginx is the chart's default).
• helm and kubectl pointing at the cluster.
• A public name for the UI, with ports 80 and 443 reachable from the internet for Let's Encrypt. The demo uses sslip.io: pnex-78-232-42-13.sslip.io resolves to 78.232.42.13, no DNS to manage.
• For devices (chart 0.3.0): a LoadBalancer Service provider (cloud LB, MetalLB, k3s ServiceLB) and a DNS name for the device endpoint (deviceEdge.host, default devices.<publicHost>), pointed at the external IP of the <release>-device-edge Service. Devices authenticate there with a client certificate (mutual TLS), which an Ingress cannot forward: they go through a dedicated nginx, never through the Ingress or a CDN.
• A CNI that enforces NetworkPolicies (Calico, Cilium...) for networkPolicy.enabled to have any effect.

curl -sfL https://get.k3s.io | sh -
kubectl get nodes
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 | bash
echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> ~/.bashrc

1. cert-manager (TLS certificates)Copy Anchor Link

cert-manager obtains and renews the Let's Encrypt certificate:

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.2 \
  -n cert-manager --create-namespace --set crds.enabled=true --wait

Then a ClusterIssuer for Let's Encrypt, which validates the domain through the ingress controller (HTTP-01). Set ingressClassName to your controller's class (traefik on k3s, nginx with ingress-nginx), and replace [email protected] with your own email:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]       # ← your email (Let's Encrypt account)
    privateKeySecretRef:
      name: letsencrypt-account
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

kubectl apply -f cluster-issuer.yaml
kubectl get clusterissuer   # READY True

2. CloudNativePG operator (PostgreSQL)Copy Anchor Link

The chart declares its database as a CloudNativePG Cluster: the operator must be installed first.

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm install cnpg cnpg/cloudnative-pg --version 0.29.1 \
  -n cnpg-system --create-namespace --wait
kubectl -n cnpg-system get pods

3. PNeX (Helm chart)Copy Anchor Link

Fetch the chart. The chart of a release tag deploys the images of that release (its appVersion); on main it follows latest.

git clone --depth 1 https://github.com/Pnex/pnex-deploy                          # main: chart 0.3.0
git clone --depth 1 --branch v0.1.0-beta.7 https://github.com/Pnex/pnex-deploy   # or the latest beta (chart 0.2.1)

git may warn that the tag "is not a commit" and that you are in "detached HEAD" state: both are harmless for an annotated release tag.

The values: the public name, the admin account and the ingress. ingress.tls points at the certificate requested from cert-manager:

publicHost: pnex-78-232-42-13.sslip.io
admin:
  email: [email protected]         # ← your email (PNeX admin login)
ingress:
  className: traefik      # nginx with ingress-nginx
  tls:
    - secretName: pnex-tls

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pnex-tls
  namespace: pnex
spec:
  secretName: pnex-tls
  dnsNames: [pnex-78-232-42-13.sslip.io]
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt

kubectl create namespace pnex && kubectl apply -f certificate.yaml
helm upgrade --install pnex ./pnex-deploy/helm/pnex -n pnex -f pnex-values.yaml
kubectl -n pnex wait --for=condition=Available deployment --all --timeout=15m
kubectl -n pnex get pods

The first start takes a few minutes (image pulls, then PostgreSQL). pnex-api restarts a few times while the database comes up: this is expected.

Device endpoint (chart 0.3.0)Copy Anchor Link

deviceEdge is on by default: a dedicated nginx behind a LoadBalancer Service that serves only device websockets (/ws/device, /ws/camera) and OTA downloads. Get its external IP and create the DNS record of devices.<publicHost>:

kubectl -n pnex get svc pnex-device-edge   # EXTERNAL-IP

• Certificate: by default the chart generates a private CA and the endpoint's certificate once (Secret <release>-device-edge-tls, kept across upgrades); every firmware pins that CA. For a public certificate, set deviceEdge.tls.existingSecret (a kubernetes.io/tls Secret, e.g. from cert-manager) and put its root in deviceCa.pem (default ISRG Root X1). One root only on ESP8266, two on ESP32; a bigger CA fails the build (build_ca_too_large).
• Single-node k3s: ServiceLB already uses 443 for the ingress controller; set deviceEdge.service.port: 4443. A port other than 443 is appended to the name compiled into firmware.

deviceEdge:
  service:
    port: 4443

4. Check and get the admin passwordCopy Anchor Link

kubectl -n pnex get certificate,ingress           # certificate READY True
curl -sI https://pnex-78-232-42-13.sslip.io/ | head -1

The admin password is generated in the cluster on first install. Read it from the pnex-secrets Secret (<release>-secrets):

kubectl -n pnex get secret pnex-secrets -o jsonpath='{.data.admin-password}' | base64 -d; echo

Sign in at https://<publicHost>/ with admin.email and this password. It only seeds the account on the first Rauthy start: change it from the account page afterwards (the Secret value is then no longer valid).

SecretsCopy Anchor Link

Every secret left empty is generated in the cluster on first install and kept across upgrades. To manage them yourself, keep them in an encrypted values file (sops) or point secrets.existingSecret at your own Secret.

kubectl -n pnex get secret pnex-secrets -o jsonpath='{.data.secrets-keys}' | base64 -d; echo

Security defaultsCopy Anchor Link

• Containers run as non-root, without privilege escalation or capabilities, with the RuntimeDefault seccomp profile and no service account token.
• Valkey requires a password; Valkey, OpenObserve and RustFS only accept PNeX pods (NetworkPolicy, needs a CNI that enforces it).
• Unauthenticated routes are rate limited. Behind a CDN, add its IP ranges to api.rateLimit.trustedProxies so limits apply per client.
• Sign-in is authorization code with PKCE only.
• The custom firmware IDE is off (customFirmware.enabled).
• Devices reach the API only through the device endpoint: TLS 1.2+, their token in the Authorization header, a client certificate issued by their organisation's CA. The endpoint adds a shared secret (secrets.edgeSecret, generated) without which the API trusts no TLS or certificate header, so a pod reaching the API Service directly cannot impersonate it. Its access log is off.
• The <release>-ws Ingress now carries browser websockets only: a device link arriving through it is refused.
• Rauthy trusts forwarded client addresses from the private ranges (the pods of the ingress controller, on any CNI). On a shared cluster, narrow rauthy.trustedProxies to your pod CIDR.

ScalingCopy Anchor Link

Set api.replicas above 1: the pods share the HTTP and WebSocket traffic and form a flow execution cluster — each organisation's flows run on exactly one pod, and move when a pod fails or drains. Rolling updates drain the old pod first. Keep replicas × (api.dbMaxConnections + 2) below PostgreSQL's max_connections.

UpgradeCopy Anchor Link

git -C pnex-deploy fetch --depth 1 origin tag v0.1.0-beta.7   # the next release
git -C pnex-deploy checkout v0.1.0-beta.7
helm upgrade pnex ./pnex-deploy/helm/pnex -n pnex -f pnex-values.yaml

image.tag overrides the chart's appVersion (e.g. main-<sha>).

Upgrading from 0.2.xCopy Anchor Link

Breaking (device security): after upgrading to chart 0.3.0, every device must be rebuilt and USB-reflashed; firmware built before is refused. On the server side, moving past 0.1.0-beta.7 also means recreating the database (see Upgrades).

• New device endpoint (deviceEdge, on by default): create the DNS record of deviceEdge.host for the <release>-device-edge Service.
• With the generated certificate, firmware now pins the device endpoint's CA instead of deviceCa.pem.
• New secret key edge-secret: with secrets.existingSecret, add it (at least 16 random characters).
• The <release>-ws Ingress now only carries browser websockets: device links arriving through it are refused.
• Re-enrol edge agents.