lumnik. / scenarios All scenarios · Français

Paperless-ngx: a list that moves while it is read is read again

Goal Read every correspondent of a Paperless-ngx archive through its API, even while someone works in it.

A real Paperless-ngx 2.13 and 100 correspondents, read ten per page.

manifests/lab-docker-compose.yml
# Paperless-ngx, a document management system on Django REST Framework, with its broker. SQLite, the default.name: lumnik-paperlessservices:  broker:    image: redis:7  paperless:    image: ghcr.io/paperless-ngx/paperless-ngx:2.13.5    depends_on: [broker]    environment:      PAPERLESS_REDIS: redis://broker:6379      PAPERLESS_ADMIN_USER: lumnik      PAPERLESS_ADMIN_PASSWORD: ${PAPERLESS_PASSWORD:-paperless-lab-password}      PAPERLESS_URL: http://host.docker.internal:${PAPERLESS_PORT:-9330}      PAPERLESS_SECRET_KEY: lab-only-secret-key    ports: ["127.0.0.1:${PAPERLESS_PORT:-9330}:8000"]   # this machine only
manifests/10-correspondents.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-paperless-a  connector_type: rest-generic  scopes: [archive]spec:  base_url: http://host.docker.internal:9330/api  auth:    kind: basic    username: lumnik    password_env: SCENARIO_PAPERLESS_A_PASSWORD  default_rate_limit: 1/s                              # a gentle reader: ten pages take ten seconds  endpoints:    - id: correspondents      path: /correspondents/?page_size=10      method: GET      pagination: { kind: drf-page }                   # follows $.next, reads $.results, checks $.count      target:        kind: hybrid        table: ext.scenario_paperless_a        unique_key: $.id        promote:          - { name: name, path: $.name, type: text }          - { name: documents, path: $.document_count, type: integer }
$ lm connector run scenario-paperless-a
run scheduled: 849bc650-cac4-4e43-a940-5f8ceb852500

$ lm run get 849bc650-cac4-4e43-a940-5f8ceb852500
ID        CONNECTOR             ENDPOINT        STATUS     IN   OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED             EVALUATED
849bc650  scenario-paperless-a  correspondents  Completed  159  159  0        0             2026-11-03 14:46  2026-11-03 14:46  yes
$ lm connector run scenario-paperless-a
run scheduled: 2b20a586-1f93-4067-93a7-9f4739abb7ee

$ lm run get 2b20a586-1f93-4067-93a7-9f4739abb7ee
ID        CONNECTOR             ENDPOINT        STATUS  IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED             EVALUATED
2b20a586  scenario-paperless-a  correspondents  Failed  20  20   0        0             2026-11-03 14:46  2026-11-03 14:46  yes
✖ run failed: Endpoint 'correspondents': the list changed while it was read, and again when read a second time (the source declared 101 rows, then 102) — rows may be on no page; refusing to report a partial extract as a complete one (run it when the source is quieter)

Limits

  • A deleted correspondent stays in the hub's table: a full read adds and updates, it does not erase.
  • The second read counts the rows read twice in IN; the upsert keeps one.
  • 100 fictional correspondents, created by the script.

Archive · documents

The correspondents live in Paperless-ngx.

Every one must be read, every night.

Paperless-ngx 2.13, 100 correspondents

The problem

The API hands the list out in pages.

  • Ten per page, numbered.
  • Meanwhile, the team works.

What is not changed

The archive stays as it is.

  • Its API, a reading account.
  • No module, no setting.

The trap

A deletion shifts the pages.

  • A correspondent on page 1 disappears.
  • Another one lands on no page.

What had to be written

pagination: { kind: drf-page }

One line.

  • Each page's count is checked.
  • If it moves, the list is read again.
  • A 27-line connector.

When it never stops

The run fails, it does not lie.

  • The list moves again on the second read.
  • The message gives both totals.

What was fixed

One gap found, one fix.

  • Before: Completed, a correspondent missing.
  • Now: read again, or refused (#1771).

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →