lumnik. / scenarios All scenarios · Français

The carrier wants its cursor in a header or the run fails

Goal Read every tracking event of a carrier whose API expects the page cursor in a request header.

A carrier's tracking API pages by cursor. It reads it from the X-Next-Cursor header, and nowhere else:

events
100
has_more
true
next_cursor
eyJvIjogMTAwfQ==

evidence: replay.log, page-1.json, requests-query.txt, requests-header.txt, parcels-by-status.txt, count.txt · transform v1

manifests/10-events-query.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-carrier-a  connector_type: rest-generic  scopes: [transport]spec:  base_url: http://host.docker.internal:9240  auth: { kind: bearer, token_env: SCENARIO_CARRIER_TOKEN }  endpoints:    - id: events      path: /v2/tracking-events      method: GET      pagination:        kind: cursor        cursor_param: cursor              # sent as ?cursor=… — the default binding        cursor_path: $.next_cursor        has_more_path: $.has_more        response_path: $.events      target:        kind: hybrid        table: ext.scenario_carrier_a        unique_key: $.event_id        promote:          - { name: parcel, path: $.parcel, type: text }          - { name: status, path: $.status, type: text }          - { name: at, path: $.at, type: timestamptz }
manifests/20-events-header.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-carrier-a  connector_type: rest-generic  scopes: [transport]spec:  base_url: http://host.docker.internal:9240  auth: { kind: bearer, token_env: SCENARIO_CARRIER_TOKEN }  endpoints:    - id: events      path: /v2/tracking-events      method: GET      pagination:        kind: cursor        cursor_binding: header            # the carrier reads its cursor from a header        cursor_param: X-Next-Cursor        cursor_path: $.next_cursor        has_more_path: $.has_more        response_path: $.events      target:        kind: hybrid        table: ext.scenario_carrier_a        unique_key: $.event_id        promote:          - { name: parcel, path: $.parcel, type: text }          - { name: status, path: $.status, type: text }          - { name: at, path: $.at, type: timestamptz }
hub
1200 events, 300 parcels
AVISE
60 parcels
EN_LIVRAISON
60 parcels
EN_TRANSIT
60 parcels
LIVRE
60 parcels
PRIS_EN_CHARGE
60 parcels

evidence: replay.log, page-1.json, requests-query.txt, requests-header.txt, parcels-by-status.txt, count.txt · transform v1

Limits

  • The refusal recognises a source that sends back the cursor it was just given. A source returning random pages would not be recognised.
  • Without a total published by the API, no witness compares the count read with the real one.
  • Simulated carrier API, synthetic events.

Transport · parcel tracking

The carrier publishes 1,200 tracking events.

The import must read them all, page after page.

Simulated carrier API

The problem

The API expects the next page in a header.

  • Sent elsewhere, it is ignored.
  • The API then returns the first page.
  • It still says there is more.

What is not changed

The API stays as it is.

  • A carrier token, stored with lm secret.
  • No work on the carrier's side.

What had to be written

cursor_binding: header
cursor_param: X-Next-Cursor

Say where the cursor goes.

One 30-line connector.

Red

Cursor misplaced: the run fails.

  • It says why, and which key to check.
  • No partial read reported complete.

Green

1,200 events, 12 pages.

  • 300 parcels, each at its latest status.
  • The cursor goes in the header on every page.

The limit

The refusal relies on the API repeating its page.

  • An API returning random pages would not be recognised.
  • The count's witness is the total the API publishes, if it publishes one.

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →