lumnik. / scenarios All scenarios · Français

FastAPI: 130 deliveries read in one line

Goal Read every delivery of an in-house FastAPI service through its API, without asking its team for an export.

A real FastAPI service on fastapi-pagination 0.16.1 and 130 fictional deliveries.

manifests/lab-docker-compose.yml
name: lumnik-scenario-fastapiservices:  deliveries:    build: ../lab    ports: ["127.0.0.1:${FASTAPI_PORT:-9340}:8000"]
manifests/10-deliveries.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-fastapi-a  connector_type: rest-generic  scopes: [logistique]spec:  base_url: http://host.docker.internal:9340  endpoints:    - id: deliveries      path: /deliveries      method: GET      pagination: { kind: fastapi }                    # ?page=1&size=50, reads $.items, stops at $.pages      target:        kind: hybrid        table: ext.scenario_fastapi_a        unique_key: $.id        promote:          - { name: reference, path: $.reference, type: text }          - { name: city, path: $.city, type: text }          - { name: parcels, path: $.parcels, type: integer }
manifests/20-deliveries-500.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-fastapi-big-a  connector_type: rest-generic  scopes: [logistique]spec:  base_url: http://host.docker.internal:9340  endpoints:    - id: deliveries      path: /deliveries      method: GET      pagination: { kind: fastapi, size: 500 }        # fastapi-pagination caps size at 100      target:        kind: hybrid        table: ext.scenario_fastapi_big_a        unique_key: $.id        promote:          - { name: reference, path: $.reference, type: text }          - { name: city, path: $.city, type: text }          - { name: parcels, path: $.parcels, type: integer }
manifests/30-deliveries-size-in-path.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-fastapi-path-a  connector_type: rest-generic  scopes: [logistique]spec:  base_url: http://host.docker.internal:9340  endpoints:    - id: deliveries      path: /deliveries?size=500                     # the page size written in the path      method: GET      pagination: { kind: fastapi }                    # ?page=1&size=50, reads $.items, stops at $.pages      target:        kind: hybrid        table: ext.scenario_fastapi_path_a        unique_key: $.id        promote:          - { name: reference, path: $.reference, type: text }          - { name: city, path: $.city, type: text }          - { name: parcels, path: $.parcels, type: integer }

Limits

  • fastapi-pagination caps size at 100 by default; the service's team may change that limit.
  • 130 fictional deliveries, created when the service starts.

Logistics · deliveries

The deliveries live in an in-house service.

The logistics team wrote it in Python.

FastAPI, fastapi-pagination 0.16.1, 130 deliveries

The problem

The API returns the list page by page.

  • page and size in the address.
  • The total and the page count in every answer.

What is not changed

The service stays as it is.

  • Its API, as the team wrote it.
  • No export, no development.

What had to be written

pagination: { kind: fastapi }

One line.

  • page=1, size=50, then the next ones.
  • Reads items, stops at pages.
  • A 22-line connector.

Quiet

130 of 130, in three requests.

  • page=1, 2, 3, fifty per page.
  • 130 rows in the hub.

Too big

500 per page: the library refuses.

  • fastapi-pagination caps size at 100.
  • Its 422 refusal reaches the run as written.
  • Nothing is ingested.

What was fixed

A gap found, a fix.

  • Before: size=500 in the path, sent twice, ignored.
  • Now: refused at lm apply (#1789).

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →