An unapproved hotfix in k8s prod stops the pipeline

Goal Block a release when a pod runs a version the release board did not approve, with an exit code a CI understands.

The catalogue of approved versions is a CSV. What really runs is in the cluster. lumnik reads each pod's image through the Kubernetes API and joins it to the catalogue:

the approved version passes;
a hotfix pushed with kubectl set image makes the gate return exit code 1: CI stops the pipeline;
after the rollback, the same command passes again.

lumnik Open
DevOps
Kubernetes
Delivery
Connector
Entity

manifests/lab-10-shop.yaml

# Three services, each image tagged with its version. run.sh builds the images (busybox, retagged) and loads them into kind.# The labels the connector reads: app (the service), environment, release-gate (what the reader lists).apiVersion: v1kind: Namespacemetadata: { name: shop }---apiVersion: v1kind: Namespacemetadata: { name: shop-staging }---apiVersion: apps/v1kind: Deploymentmetadata: { name: billing-api, namespace: shop }spec:  replicas: 1  selector: { matchLabels: { app: billing-api } }  template:    metadata: { labels: { app: billing-api, environment: production, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: api, image: "lumnik.local/billing-api:2.3.7", command: ["sleep", "infinity"] }---apiVersion: apps/v1kind: Deploymentmetadata: { name: catalog-api, namespace: shop }spec:  replicas: 1  selector: { matchLabels: { app: catalog-api } }  template:    metadata: { labels: { app: catalog-api, environment: production, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: api, image: "lumnik.local/catalog-api:5.8.1", command: ["sleep", "infinity"] }---# A release candidate in staging: not approved, and not production — the gate lets it through.apiVersion: apps/v1kind: Deploymentmetadata: { name: customer-portal, namespace: shop-staging }spec:  replicas: 1  selector: { matchLabels: { app: customer-portal } }  template:    metadata: { labels: { app: customer-portal, environment: staging, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: web, image: "lumnik.local/customer-portal:9.1.0-rc2", command: ["sleep", "infinity"] }

manifests/lab-30-reader.yaml

# What lumnik may do in the cluster: read pods, cluster-wide. Nothing else.apiVersion: v1kind: Namespacemetadata: { name: release-gate-reader }---apiVersion: v1kind: ServiceAccountmetadata: { name: release-gate-reader, namespace: release-gate-reader }---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: { name: release-gate-reader }rules:  - apiGroups: [""]    resources: [pods]    verbs: [get, list]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: { name: release-gate-reader }roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: release-gate-reader }subjects: [{ kind: ServiceAccount, name: release-gate-reader, namespace: release-gate-reader }]---# A long-lived token for the hub's connector secret (projected tokens expire within the hour).apiVersion: v1kind: Secretmetadata:  name: release-gate-reader-token  namespace: release-gate-reader  annotations: { kubernetes.io/service-account.name: release-gate-reader }type: kubernetes.io/service-account-token

$ kubectl --context kind-lumnik-memory get pods -A -l release-gate -o "custom-columns=SERVICE:.metadata.labels.app,ENV:.metadata.labels.environment,IMAGE:.status.containerStatuses[0].image"
SERVICE           ENV          IMAGE
customer-portal   staging      lumnik.local/customer-portal:9.1.0-rc2
billing-api       production   lumnik.local/billing-api:2.3.7
catalog-api       production   lumnik.local/catalog-api:5.8.1

approved-versions.csv

service,owner,approved_version
billing-api,payments-on-call,2.3.7
catalog-api,commerce-on-call,5.8.1
customer-portal,web-on-call,9.0.4

manifests/10-approved-versions.yaml

# The catalogue of approved versions: a CSV the release board maintains, outside the cluster.apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: approved-versions  connector_type: csv-file  scopes: [devops-release-gate]spec:  transport:    kind: local    path: /tmp/lumnik-scenario/devops-release-gate/approved-versions.csv  parser:    has_header: true  after_process: none

manifests/20-pods.yaml

# The cluster's own API, read-only: the pods that run a service, and the image each one runs.# The token and the CA come from the release-gate-reader ServiceAccount (lab-30-reader.yaml): lm secret set.apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: k8s-pods  connector_type: rest-generic  scopes: [devops-release-gate]spec:  base_url: https://kubernetes.default.svc  auth: { kind: bearer, token_env: RELEASE_READER_TOKEN }  tls: { ca_secret: CLUSTER_CA }  defaults:    incremental: { mode: full }  endpoints:    - id: pods      method: GET      path: /api/v1/pods?labelSelector=release-gate%3Dtrue      pagination: { kind: cursor, cursor_param: continue, cursor_path: $.metadata.continue, response_path: $.items }      target:        kind: hybrid        table: connector.release_pods        unique_key: $.metadata.uid        promote:          - { name: pod_uid,     path: "$.metadata.uid",                     type: text }          - { name: service,     path: "$.metadata.labels.app",              type: text }          - { name: environment, path: "$.metadata.labels.environment",      type: text }          - { name: started_at,  path: "$.metadata.creationTimestamp",       type: timestamptz }          - { name: image,       path: "$.status.containerStatuses[0].image", type: text }          - { name: deployed_version, type: text }   # filled by image-version.kts      mapping:        cell:          - kotlin: { script: hooks:image-version.kts }

manifests/image-version.kts

// The version a pod really runs: its image's tag (lumnik.local/billing-api:2.3.7 → 2.3.7).@Suppress("UNCHECKED_CAST")val cells = bindings["cells"] as MutableMap<String, Any?>val image = cells["image"] as String?cells["deployed_version"] = image?.substringAfterLast('/')?.substringAfter(':', "")?.ifEmpty { null }

$ lm apply -f manifests/10-approved-versions.yaml
Applied connector "approved-versions" — 1 endpoint

$ lm apply -f manifests/20-pods.yaml
Applied connector "k8s-pods" — 1 endpoint
! 20-pods.yaml — 0 error(s), 1 warning(s)

  WARN     spec.endpoints[0].pagination.has_more_path
          no has_more_path: the cursor decides — the read stops on an empty cursor or an empty page
          → Declare has_more_path if the API sends a more-pages flag


$ lm connector run approved-versions
run scheduled: 1792aa2c-a9a5-46c3-ac4a-fe3186b0fb3b

$ lm run get 1792aa2c-a9a5-46c3-ac4a-fe3186b0fb3b
ID        CONNECTOR          ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
1792aa2c  approved-versions  default   Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06

$ lm connector run k8s-pods
run scheduled: f73a6843-4e26-44a2-962b-f5df3d0b2c14

$ lm run get f73a6843-4e26-44a2-962b-f5df3d0b2c14
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
f73a6843  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

manifests/30-deployment-compliance.yaml

kind: Entityname: DeploymentCompliancedescription: "Each pod a service ran, the version of its image, and the version the release board approved for that service."match: [pod_uid]scopes: [devops-release-gate]sources:  - table: release_pods    map:      pod_uid: pod_uid      service: service      environment: environment      deployed_version: deployed_version      started_at: started_atjoins:  - table: t_approved_versions    from: service    to: service    map:      owner: owner      approved_version: approved_version

$ lm entity apply -f manifests/30-deployment-compliance.yaml
Applied entity "DeploymentCompliance" → connector."v_deploymentcompliance"

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[]

$ kubectl --context kind-lumnik-memory -n shop set image deploy/billing-api api=lumnik.local/billing-api:2.4.0-hotfix
deployment.apps/billing-api image updated

$ lm connector run k8s-pods
run scheduled: 7e0558c7-aec1-4865-9a9d-f1243040d98a

$ lm run get 7e0558c7-aec1-4865-9a9d-f1243040d98a
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
7e0558c7  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "1b86c1e6-3538-47ad-b375-38995784ca70",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.4.0-hotfix",
    "started_at": "2026-10-30T09:06:13Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[
  {
    "service": "billing-api",
    "owner": "payments-on-call",
    "deployed_version": "2.4.0-hotfix",
    "approved_version": "2.3.7"
  }
]

$ kubectl --context kind-lumnik-memory -n shop rollout undo deploy/billing-api
deployment.apps/billing-api rolled back
Warning: resource deployments/billing-api was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.

$ lm connector run k8s-pods
run scheduled: 0f27922d-038c-41df-91ec-5469cde29f9c

$ lm run get 0f27922d-038c-41df-91ec-5469cde29f9c
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
0f27922d  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "1b86c1e6-3538-47ad-b375-38995784ca70",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.4.0-hotfix",
    "started_at": "2026-10-30T09:06:13Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "daa99f47-0867-4fca-a2a8-cc78522350a1",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:18Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[]

service=billing-api: owner=payments-on-call deployed=2.3.7 approved=2.3.7
service=catalog-api: owner=commerce-on-call deployed=5.8.1 approved=5.8.1

evidence: replay.log, approved-statuses.txt, approved-violations.json, hotfix-statuses.txt, hotfix-violations.json, rollback-statuses.txt, rollback-violations.json, rollback-entity.json, summary.env · transform v1

Limits

The gate sees what the hub has read: the pipeline runs the cluster read just before the gate.
The version is the image tag. An image pushed under an already-approved tag is not detected: comparing digests would need another column.
One container per pod is read (the first).
Demo kind cluster, images built for the replay.

DevOps · release

A hotfix reaches production without approval.

The pipeline has to see it.

Demo kind cluster

The problem

Two truths, two places.

Approved versions: the board's file.
Running versions: the cluster.
No tool compares them before shipping.

What is not modified

lumnik reads; it changes nothing.

A service account that can only list pods.
The CSV stays where it is.

What had to be written

path: /api/v1/pods?labelSelector=…
joins:
  - table: t_approved_versions

One read, one join.

4 files, 72 lines, and an 18-line shell gate.

Red

billing-api: 2.4.0-hotfix, approved 2.3.7.

The pipeline stops.
The violation names the service and its on-call team.

Green

Rollback: the same command passes.

The hub keeps every pod it saw.
The gate reads the newest.

The limit

The gate sees what the hub has read.

A hotfix made after the last read is not seen yet.
The pipeline runs the read just before the gate.

Try it

The scenario replays on kind.

Two connectors, a script, an entity, a jq gate: every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →