lumnik. / scenarios All scenarios · Français

The partner's API loses a page and the run does not lie

Goal Never let a run call itself complete when the source served fewer rows than it declares.

A partner publishes 2,400 rows and their count. One night, a page of 100 vanishes, the count stays right.

manifests/10-partner.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-partner-a  connector_type: rest-generic  scopes: [partners]spec:  base_url: http://host.docker.internal:9220  endpoints:    - id: partners      path: /partners      method: GET      pagination:        kind: offset        offset_param: offset        limit_param: limit        limit: 500        response_path: "$.data"        total_path: "$.total"      target:        kind: hybrid        table: ext.scenario_partner_a        unique_key: "$.id"        promote:          - { name: name, path: "$.name", type: text }          - { name: city, path: "$.city", type: text }
$ lm connector run scenario-partner-a
run scheduled: c286c444-9b4d-4ec4-b544-d0734e0c9ab1

$ lm run get c286c444-9b4d-4ec4-b544-d0734e0c9ab1
ID        CONNECTOR           ENDPOINT  STATUS     IN    OUT   SKIPPED  DEAD-LETTERS  STARTED           ENDED
c286c444  scenario-partner-a  partners  Completed  2400  2400  0        0             2026-11-03 17:11  2026-11-03 17:11
$ lm connector run scenario-partner-a
run scheduled: 8b414680-f5ed-47a9-ba29-bf19a2c91c7e

$ lm run get 8b414680-f5ed-47a9-ba29-bf19a2c91c7e
ID        CONNECTOR           ENDPOINT  STATUS  IN    OUT   SKIPPED  DEAD-LETTERS  STARTED           ENDED
8b414680  scenario-partner-a  partners  Failed  2000  2000  0        0             2026-11-03 17:11  2026-11-03 17:11
✖ run failed: Endpoint 'partners': the source declared 2400 rows but this run returned 2300 — refusing to report a partial extract as a complete one
$ lm connector run scenario-partner-a
run scheduled: d7079df1-b8b4-4738-9934-8ed65dfa629d

$ lm run get d7079df1-b8b4-4738-9934-8ed65dfa629d
ID        CONNECTOR           ENDPOINT  STATUS     IN    OUT   SKIPPED  DEAD-LETTERS  STARTED           ENDED
d7079df1  scenario-partner-a  partners  Completed  2400  2400  0        0             2026-11-03 17:11  2026-11-03 17:11

$ lm run list --connector scenario-partner-a
ID        CONNECTOR           ENDPOINT  STATUS     IN    OUT   SKIPPED  DEAD-LETTERS  STARTED           ENDED
d7079df1  scenario-partner-a  partners  Completed  2400  2400  0        0             2026-11-03 17:11  2026-11-03 17:11
8b414680  scenario-partner-a  partners  Failed     2000  2000  0        0             2026-11-03 17:11  2026-11-03 17:11
c286c444  scenario-partner-a  partners  Completed  2400  2400  0        0             2026-11-03 17:11  2026-11-03 17:11

Limits

  • The witness is only as good as a real row count published by the source.
  • The check fires when pagination ends: rows already written stay.
  • A source that publishes no total has no witness.
  • Partner API stand-in; synthetic data.

Partner · API

The partner publishes its rows and their count.

2,400, page by page.

Simulated API, three nights

The problem

A page vanishes.

  • 100 rows fewer.
  • The declared count does not move.

What is not changed

The API stays as it is.

  • Its total, published with each page.
  • No check to add on its side.

The trap

A green run over a partial list.

  • The paging ends normally.
  • 100 rows are missing.

What had to be written

total_path: "$.total"

One line.

  • The source's total is the witness.
  • Read ≠ declared: the run fails.
  • A 28-line scenario, one manifest.

The faulty night

Failed, with both numbers.

  • 2,300 served, 2,400 declared.
  • The endpoint named.

The limit

It needs a real total.

  • A source with no total has no witness.

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →