k8s forgot its events and lumnik still knows who crashed last night

Goal Ask which deployments crashed in a loop after Kubernetes erased their events, and get the list kubectl gave during the incident.

Kubernetes keeps its events for one hour. A deployment repaired at dawn has no history by morning:

a new pod, Running, 0 restarts;
no Warning event left.

lumnik reads the events while they exist and keeps them; Ask still answers when kubectl no longer knows.

lumnik Pro
DevOps
Kubernetes
Post-mortem
Connector
Entity

manifests/kind-cluster.yaml

# A kind cluster that forgets its events after 3 minutes instead of one hour (kube-apiserver --event-ttl):# the same forgetting a production cluster does, sped up so the stand can watch it happen.kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:  - role: control-plane    kubeadmConfigPatches:      - |        kind: ClusterConfiguration        apiServer:          extraArgs:            event-ttl: 3m

manifests/lab-10-workloads.yaml

# Three deployments. Each failure is produced by Kubernetes itself, nothing is faked.# payment-api: exits 1 after five seconds — restarted, then CrashLoopBackOff. run.sh repairs it later (lab-20).apiVersion: apps/v1kind: Deploymentmetadata: { name: payment-api, namespace: payments, labels: { team: team-payments } }spec:  replicas: 1  selector: { matchLabels: { app: payment-api } }  template:    metadata: { labels: { app: payment-api, team: team-payments } }    spec:      containers:        - name: api          image: busybox:1.36          command: ["sh", "-c", "sleep 5; exit 1"]---# billing-api: the same crash, never repaired — still in CrashLoopBackOff when the question is asked.apiVersion: apps/v1kind: Deploymentmetadata: { name: billing-api, namespace: billing, labels: { team: team-billing } }spec:  replicas: 1  selector: { matchLabels: { app: billing-api } }  template:    metadata: { labels: { app: billing-api, team: team-billing } }    spec:      containers:        - name: api          image: busybox:1.36          command: ["sh", "-c", "sleep 5; exit 1"]---# catalog-api: healthy from the start, never restarted.apiVersion: apps/v1kind: Deploymentmetadata: { name: catalog-api, namespace: catalog, labels: { team: team-catalog } }spec:  replicas: 1  selector: { matchLabels: { app: catalog-api } }  template:    metadata: { labels: { app: catalog-api, team: team-catalog } }    spec:      containers:        - name: api          image: busybox:1.36          command: ["sh", "-c", "sleep infinity"]

manifests/lab-30-reader.yaml

# What lumnik may do in the cluster: read events, cluster-wide. Nothing else.apiVersion: v1kind: Namespacemetadata: { name: lumnik-reader }---apiVersion: v1kind: ServiceAccountmetadata: { name: lumnik-reader, namespace: lumnik-reader }---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: { name: lumnik-reader }rules:  - apiGroups: [""]    resources: [events]    verbs: [get, list]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: { name: lumnik-reader }roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: lumnik-reader }subjects: [{ kind: ServiceAccount, name: lumnik-reader, namespace: lumnik-reader }]---# A long-lived token for the hub's connector secret (projected tokens expire within the hour).apiVersion: v1kind: Secretmetadata:  name: lumnik-reader-token  namespace: lumnik-reader  annotations: { kubernetes.io/service-account.name: lumnik-reader }type: kubernetes.io/service-account-token

$ kubectl --context kind-lumnik-memory get pods -A -l team
NAMESPACE   NAME                           READY   STATUS    RESTARTS      AGE
billing     billing-api-7d7c7d98c4-4c46p   1/1     Running   3 (23s ago)   53s
catalog     catalog-api-74db49b46f-nrl9f   1/1     Running   0             53s
payments    payment-api-5b8cc46f9f-9rcxr   1/1     Running   3 (24s ago)   53s

manifests/10-cluster.yaml

# The cluster's own API, read-only: its Warning events about pods.# Each run keeps what it read; an event the cluster forgets afterwards stays in the hub.# The token and the CA come from the lumnik-reader ServiceAccount (lab-30-reader.yaml): lm secret set.apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: k8s-events  connector_type: rest-generic  scopes: [devops-event-memory]spec:  base_url: https://kubernetes.default.svc  auth: { kind: bearer, token_env: K8S_READER_TOKEN }  tls: { ca_secret: CLUSTER_CA }  defaults:    incremental: { mode: full }  endpoints:    - id: warnings      method: GET      path: /api/v1/events?fieldSelector=type%3DWarning,involvedObject.kind%3DPod      pagination: { kind: cursor, cursor_param: continue, cursor_path: $.metadata.continue, response_path: $.items }      target:        kind: hybrid        table: connector.k8s_warnings        unique_key: $.metadata.uid        promote:          - { name: event_uid,  path: "$.metadata.uid",            type: text }          - { name: namespace,  path: "$.involvedObject.namespace", type: text }          - { name: pod,        path: "$.involvedObject.name",      type: text }          - { name: reason,     path: "$.reason",                   type: text }          - { name: count,      path: "$.count",                    type: integer }          - { name: first_seen, path: "$.firstTimestamp",           type: timestamptz }          - { name: last_seen,  path: "$.lastTimestamp",            type: timestamptz }          - { name: message,    path: "$.message",                  type: text }          - { name: workload_key, type: text }       # filled by workload-key.kts      mapping:        cell:          - kotlin: { script: hooks:workload-key.kts }

manifests/workload-key.kts

// The deployment an event is about: namespace/deployment. A Deployment's pod is named// <deployment>-<replicaset hash>-<pod suffix>, and a new pod keeps the deployment, not the name.@Suppress("UNCHECKED_CAST")val cells = bindings["cells"] as MutableMap<String, Any?>val deployment = (cells["pod"] as String?)?.split("-")?.dropLast(2)?.joinToString("-")cells["workload_key"] = if (deployment.isNullOrEmpty()) null else "${cells["namespace"]}/$deployment"

$ lm apply -f manifests/10-cluster.yaml
Applied connector "k8s-events" — 1 endpoint
! 10-cluster.yaml — 0 error(s), 1 warning(s)

  WARN     spec.endpoints[0].pagination.has_more_path
          no has_more_path: the cursor decides — the read stops on an empty cursor or an empty page
          → Declare has_more_path if the API sends a more-pages flag

$ lm connector run k8s-events
run scheduled: 3becfb6f-fe7e-441e-a3ea-6e22d74e993b

$ lm run get 3becfb6f-fe7e-441e-a3ea-6e22d74e993b
ID        CONNECTOR   ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
3becfb6f  k8s-events  warnings  Completed  2   2    0        0             2026-10-29 23:53  2026-10-29 23:53
• script hooks:workload-key.kts  sha256 927207031e62a6e618fb9ae870f2cf6052e9ef9cbd0865af00b86d50de2f5b1e

manifests/lab-20-fix-payment.yaml

# The fix the on-call engineer ships: payment-api stops crashing. A new pod replaces the old one —# Running, 0 restarts — and the old pod's events expire with the cluster's event TTL.apiVersion: apps/v1kind: Deploymentmetadata: { name: payment-api, namespace: payments, labels: { team: team-payments } }spec:  replicas: 1  selector: { matchLabels: { app: payment-api } }  template:    metadata: { labels: { app: payment-api, team: team-payments } }    spec:      containers:        - name: api          image: busybox:1.36          command: ["sh", "-c", "sleep infinity"]

$ kubectl --context kind-lumnik-memory get pods -n payments
NAME                           READY   STATUS    RESTARTS   AGE
payment-api-56f4fbb694-pmj7w   1/1     Running   0          2m43s

$ kubectl --context kind-lumnik-memory get events -n payments --field-selector type=Warning
No resources found in payments namespace.

$ lm connector run k8s-events
run scheduled: 7407cb9f-b028-456a-8298-420e75eaa8cb

$ lm run get 7407cb9f-b028-456a-8298-420e75eaa8cb
ID        CONNECTOR   ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
7407cb9f  k8s-events  warnings  Completed  1   1    0        0             2026-10-29 23:56  2026-10-29 23:56
• script hooks:workload-key.kts  sha256 927207031e62a6e618fb9ae870f2cf6052e9ef9cbd0865af00b86d50de2f5b1e

manifests/20-warning.yaml

kind: Entityname: KubernetesWarninglabel: Kubernetes Warning event about a pod, kept after the cluster forgot itdescription: "One Warning event the cluster recorded about a pod of a deployment, read from the Kubernetes API and kept by the hub after Kubernetes expired it. reason BackOff means the container crashed and was restarted in a loop; count is how many times it repeated."match: [event_uid]scopes: [devops-event-memory]columns:  event_uid:    { label: "The event's identifier in the cluster" }  workload_key: { label: "namespace/deployment — the deployment whose pod the event is about; it survives the pod" }  pod:          { label: "The pod the event is about; a repaired deployment runs a new pod" }  reason:       { label: "Why Kubernetes warned: BackOff (restarted in a loop), Unhealthy, FailedMount…" }  count:        { label: "How many times this event repeated" }  first_seen:   { label: "When it was first seen" }  last_seen:    { label: "When it was last seen" }  message:      { label: "Kubernetes' own message" }sources:  - table: k8s_warnings    map:      event_uid: event_uid      workload_key: workload_key      pod: pod      reason: reason      count: count      first_seen: first_seen      last_seen: last_seen      message: message

$ lm entity apply -f manifests/20-warning.yaml
Applied entity "KubernetesWarning" → connector."v_kuberneteswarning"

$ lm ask --scope devops-event-memory --lang fr "Quels déploiements ont eu un événement BackOff ?"
Il y a 2 déploiements qui ont eu un événement BackOff :

1. billing/billing-api
2. payments/payment-api

Total de clés : 2
  data as of: 2026-10-29T22:56:31.363905Z (last completed run)

  WORKLOAD_KEY
  billing/billing-api
  payments/payment-api
  2 rows returned

  SQL: SELECT DISTINCT "workload_key" FROM "connector"."v_kuberneteswarning" WHERE "reason" = 'BackOff'
LIMIT 500

$ lm ask --scope devops-event-memory --lang fr -o json "Quels déploiements ont eu un événement BackOff ?"
{
  "answer": "Il y a 2 déploiements qui ont eu un événement BackOff :\n\n1. billing/billing-api\n2. payments/payment-api\n\nTotal de clés : 2",
  "sql": "SELECT DISTINCT \"workload_key\" FROM \"connector\".\"v_kuberneteswarning\" WHERE \"reason\" = 'BackOff'\nLIMIT 500",
  "rowCount": 2,
  "columns": [
    "workload_key"
  ],
  "rows": [
    {
      "workload_key": "billing/billing-api"
    },
    {
      "workload_key": "payments/payment-api"
    }
  ],
  "dataAsOf": "2026-10-29T22:56:31.363905Z",
  "otherDoors": [],
  "disputed": [],
  "disputedTruncated": false,
  "abstained": false,
  "understoodAs": null
}

Limits

The model translates the question; "redémarré en boucle" makes it abstain. A column written by the script would make the business wording possible.
The key assumes a Deployment's pod (<deployment>-<hash>-<suffix>): a StatefulSet or a bare pod gets no key.
The memory starts with the first run: an event expired before it was read is not recovered.
Demo kind cluster: the forgetting is real, sped up to 3 minutes.

DevOps · post-mortem

9am. Everything is green. Who crashed last night?

On-call prepares the post-mortem.

kubectl only shows the present.

Demo kind cluster

The problem

Kubernetes erases its events after one hour.

A repaired pod is a new pod: 0 restarts.
Its Warning events expire.
The post-mortem's question has no answer left in the cluster.

The lab

Two deployments crash in a loop.

payment-api is repaired.
billing-api still crashes.
The cluster forgets its events in 3 minutes instead of one hour.

What had to be written

unique_key: $.metadata.uid

cells["workload_key"] =
  "$namespace/$deployment"

One key per event, one per deployment.

The hub keeps every event it read. The scenario: 3 files, 69 lines.

After the forgetting

kubectl: No resources found.

payment-api: Running, 0 restarts.
The second run reads one event out of two.

The answer

"Quels déploiements ont eu un événement BackOff?"

payments/payment-api, forgotten by the cluster.
billing/billing-api, still failing.

The proof

The kubectl witness is taken during the incident.

Same list, or the scenario fails.
6 times out of 6, the same two deployments.

Try it

The scenario replays on kind.

One connector, one script, one entity, one question: every step is published with its output.

This scenario needs lumnik Pro.

See the editions →