lumnik. / scenarios All scenarios · Français

An order changed during the read is not lost

Goal Read only what changed each night, without losing what changes during the read.

An order API can answer « what changed since this date ». lumnik reads it in full once, then each night only what moved.

manifests/10-orders.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-incremental-b  connector_type: rest-generic  scopes: [sales]spec:  base_url: http://host.docker.internal:8096  endpoints:    - id: orders      path: /api/orders      method: GET      pagination:        kind: offset        offset_param: offset        limit_param: limit        limit: 50        response_path: "$.data"      incremental:        mode: watermark        param: updated_since        watermark_path: "$.updated_at"      target:        kind: hybrid        table: ext.incremental_orders_b        unique_key: "$.id"        promote:          - { name: customer, path: "$.customer", type: text }          - { name: status, path: "$.status", type: text }          - { name: amount, path: "$.amount", type: decimal }          - { name: updated_at, path: "$.updated_at", type: timestamptz }
$ lm connector run scenario-incremental-b
run scheduled: 2917bdc6-1521-488a-867a-c534c044031c

$ lm run get 2917bdc6-1521-488a-867a-c534c044031c
ID        CONNECTOR               ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
2917bdc6  scenario-incremental-b  orders    Completed  2   2    0        0             2026-11-03 17:08  2026-11-03 17:08
$ lm connector run scenario-incremental-b
run scheduled: ac6d8527-718f-4375-90a9-f9238bcc0190

$ lm run get ac6d8527-718f-4375-90a9-f9238bcc0190
ID        CONNECTOR               ENDPOINT  STATUS     IN   OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
ac6d8527  scenario-incremental-b  orders    Completed  122  122  0        0             2026-11-03 17:08  2026-11-03 17:08
$ lm connector run scenario-incremental-b
run scheduled: 5cb57a7d-542f-4579-913c-0e7ea7700f49

$ lm run get 5cb57a7d-542f-4579-913c-0e7ea7700f49
ID        CONNECTOR               ENDPOINT  STATUS     IN   OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
5cb57a7d  scenario-incremental-b  orders    Completed  120  120  0        0             2026-11-03 17:08  2026-11-03 17:08

Limits

  • The margin re-reads the orders changed in that window; they are updated, not duplicated.
  • Works when updated_at carries a time zone; a version number or a date alone is sent as is.
  • Simulated API, synthetic orders.

Sales · order API

The API says what changed.

Each night, only that.

300 orders, simulated API

The problem

An order changes during the read.

  • Already read, it moves again.
  • The next watermark lands after it.

What is not changed

The API stays as it is.

  • Its updated_since parameter.
  • No trigger to add.

The trap

A change between two nights.

  • Too late for this run.
  • Too early for the next.

What had to be written

incremental: { mode: watermark }

One line.

  • The watermark pulled back by a margin.
  • What is read again is updated.
  • A 31-line scenario, one manifest.

The result

SO-00101 arrives invoiced.

  • With the next run.
  • No duplicate.

The limit

It needs a date with a time zone.

  • A version number is sent as it is.

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →