Kubernetes and k3s

One Kubernetes Deployment is one Fleetwide deployment. The pod runs the supervisor image with the enrollment key from a Secret and is known by its pod. Kubernetes replaces pods rather than restarting them, so mark the Fleetwide deployment ephemeral with a timer and the console removes the record of a pod that has gone. Fleetwide updates what runs inside the pod; the Deployment's own rollouts are only for changing this file.

With kubectl. Put the key in a Secret and apply one file:

kubectl create secret generic my-app-fleetwide --from-literal=key='fw1.…'

# deployment.yaml — one Kubernetes Deployment per Fleetwide deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels: { app: my-app }
  template:
    metadata:
      labels: { app: my-app }
    spec:
      containers:
        - name: my-app
          image: public.ecr.aws/fleetwide/slim:latest
          env:
            - name: FLEETWIDE_KEY
              valueFrom: { secretKeyRef: { name: my-app-fleetwide, key: key } }
          ports:
            - containerPort: 8080
          volumeMounts:
            - { name: state, mountPath: /var/lib/fleetwide }
      volumes:
        - name: state
          emptyDir: {}

kubectl apply -f deployment.yaml
kubectl get pods -l app=my-app      # each pod enrolls and shows up on the fleet page

With Helm. The same thing as a chart with one template and five values, so the key never lives in a file you commit:

# values.yaml
key: ""                                   # enrollment key from the console (fw1.…)
deployment: web
image: public.ecr.aws/fleetwide/slim:latest
port: 8080
replicas: 2

# templates/deployment.yaml
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Release.Name }}-fleetwide
stringData:
  key: {{ .Values.key | quote }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels: { app: {{ .Release.Name }} }
  template:
    metadata:
      labels: { app: {{ .Release.Name }} }
    spec:
      containers:
        - name: app
          image: {{ .Values.image }}
          env:
            - name: FLEETWIDE_KEY
              valueFrom: { secretKeyRef: { name: {{ .Release.Name }}-fleetwide, key: key } }
          ports:
            - containerPort: {{ .Values.port }}
          volumeMounts:
            - { name: state, mountPath: /var/lib/fleetwide }
      volumes:
        - name: state
          emptyDir: {}

helm install my-app ./my-app-chart --set key='fw1.…'
# later: helm upgrade my-app ./my-app-chart --set replicas=3

Map the ports your app listens on, as you would for your own image. An emptyDir is enough for the state volume: a new pod enrolls and pulls the release again. If you want the orchestrator's probes to follow the supervisor, switch readiness on for the fleet and point readinessProbe and livenessProbe at /fleetwide/ready and /fleetwide/live on port 9100 (see fleetwide.yaml).

k3s. The same files work unchanged on k3s, including on arm64 devices such as a Raspberry Pi or Jetson:

sudo k3s kubectl create secret generic my-app-fleetwide --from-literal=key='fw1.…'
sudo k3s kubectl apply -f deployment.yaml

On a device that reboots, keep the state on disk so the container resumes its release without the console. k3s ships the local-path storage class:

# a device that reboots: keep the state on disk instead of an emptyDir
      volumes:
        - name: state
          persistentVolumeClaim: { claimName: my-app-state }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: my-app-state }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-path
  resources: { requests: { storage: 2Gi } }

Node images. Tested on Amazon Linux 2023, Bottlerocket and Fargate. Bottlerocket enforces SELinux and does not let a container write security.selinux labels; the supervisor skips the labels an image carries (the host labels files itself) and logs one warning per release.

Where root is not allowed in the cluster, use the embedded runtime: the same Deployment, with your own image in place of the supervisor image and a non-root securityContext.

← Install on a customer Rolling updates →