lumnik. / scenarios All scenarios · Français

The catalogue leaves for the partner in the partner's format

Goal Deliver the product catalogue to a partner in the format it imposes, with no conversion code.

Two CSV files from an ERP, products and categories, are joined in the hub and delivered to S3 in a partner's format.

manifests/10-products.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: import-export-products-b  connector_type: csv-file  scopes: [catalogue]spec:  transport:    kind: local    path: /tmp/lumnik-scenario/import-export/b/m_product.csv  parser:    has_header: true  after_process: none  mapping:    dataset:      - require-headers:          columns: [m_product_id, m_product_category_id, name, description, isactive]      - enrich-context:          record_type: ARTICLE          source_system: COMPIERE          catalogue_version: "1.5"          catalogue_date: "2026-10-15"    cell:      - parse-number: { column: m_product_id, integer: true }      - parse-number: { column: m_product_category_id, integer: true }      - clean:          columns: [name, description]          strip: true          remove-control-chars: true          normalize-unicode: NFC      - strip-invisible: { columns: [name] }      - upper: { columns: [name] }      - replace-smart-quotes: { columns: [description] }      - lookup:          column: isactive          table: { "Y": "1", "N": "0" }          on-missing: null
manifests/20-product-categories.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: import-export-product-categories-b  connector_type: csv-file  scopes: [catalogue]spec:  transport:    kind: local    path: /tmp/lumnik-scenario/import-export/b/m_product_category.csv  parser:    has_header: true  after_process: none  mapping:    dataset:      - require-headers:          columns: [m_product_category_id, name, description, isactive]    cell:      - parse-number: { column: m_product_category_id, integer: true }      - clean:          columns: [name, description]          strip: true          remove-control-chars: true          normalize-unicode: NFC      - strip-invisible: { columns: [name] }      - replace-smart-quotes: { columns: [name, description] }      - decode-html-entities: { columns: [name, description] }      - lookup:          column: isactive          table: { "Y": "1", "N": "0" }          on-missing: null
manifests/30-product-with-category.yaml
kind: Entityname: ProductWithCategoryBdescription: Products enriched with their category before delivery to S3.match: [product_id]scopes: [catalogue]sources:  - table: t_import_export_products_b    map:      record_type: record_type      source_system: source_system      catalogue_version: catalogue_version      catalogue_date: catalogue_date      m_product_id: product_id      m_product_category_id: category_id      name: product_name      description: product_description      isactive: product_activejoins:  - table: t_import_export_product_categories_b    from: category_id    to: m_product_category_id    map:      name: category_name      description: category_description      isactive: category_active
manifests/40-product-catalogue-s3.yaml
apiVersion: apps.lumnik.io/v1kind: Exportmetadata:  name: import-export-product-catalogue-bspec:  scope: catalogue  source:    entity: ProductWithCategoryB  columns:    - name: record_type      as: TYPE_ENREGISTRE    - name: source_system      as: SYSTEME_SOURCE    - name: catalogue_version      as: VERSION_FORMAT      format:        - format-number: { decimal-separator: ",", scale: 2 }    - name: catalogue_date      as: DATE_CATALOGUE      format:        - format-date: { pattern: dd/MM/yyyy }    - name: product_id      as: ID_ARTICLE    - name: product_name      as: REFERENCE_ARTICLE    - name: product_description      as: DESCRIPTION_ARTICLE    - name: product_active      as: STATUT_ARTICLE      format:        - lookup: { table: { "1": A, "0": I }, on-missing: error }    - name: category_id      as: REF_CATEGORIE      format:        - mask: { keep-last: 4 }    - name: category_name      as: LIBELLE_CATEGORIE    - name: category_description      as: DESCRIPTION_CATEGORIE  filter:    - column: category_active      equals: "1"  order: [product_id]  format:    kind: csv    delimiter: ";"  destination:    kind: s3    bucket: import-export    prefix: catalogue/b    filename: products.csv    region: us-east-1    endpoint: http://import-export-minio:9000    access_key_env: IMPORT_EXPORT_S3_ACCESS_KEY    secret_key_env: IMPORT_EXPORT_S3_SECRET_KEY  mode: snapshot
header
"TYPE_ENREGISTRE";"SYSTEME_SOURCE";"VERSION_FORMAT";"DATE_CATALOGUE";"ID_ARTICLE";"REFERENCE_ARTICLE";"DESCRIPTION_ARTICLE";"STATUT_ARTICLE";"REF_CATEGORIE";"LIBELLE_CATEGORIE";"DESCRIPTION_CATEGORIE"
first row
"ARTICLE";"COMPIERE";"1,50";"15/10/2026";"218";"294508";"FILTRE A CARTOUCHE MISTRAL 880 Ø 400";"A";"***0018";"Filtration à cartouche";
check
38300 products from active categories exported in the partner's Maximus format; headers are renamed, version and date are formatted, statuses are recoded, internal category ids are masked, and the receipt proves the exact delivered bytes.

evidence: replay.log, delivered-head.csv · transform v1

Limits

  • Masking a predictable identifier is pseudonymisation, not encryption; if the partner must not correlate categories, drop the column.
  • Snapshot mode rewrites the whole object on every run.
  • Synthetic data.

Catalogue · partner

A partner wants the catalogue.

In its format, not ours.

Semicolons, decimal commas, its own headers and codes.

Synthetic catalogue

The problem

One conversion script per partner.

  • The ERP exports products and categories separately.
  • The partner wants one line per item, category included.
  • Its statuses and dates are not the ERP's.

Nothing is changed

The ERP exports as before.

  • Two CSV files, as they are.
  • The hub joins and formats.
  • No script to maintain.

What had to be written

- name: category_id
  as: REF_CATEGORIE
  format:
    - mask: { keep-last: 4 }

The internal reference is masked.

A 149-line scenario, four declaration files.

What the partner receives

38,300 items, in its format.

  • Renamed headers.
  • Version 1,50 and date 15/10/2026.
  • Status A or I, reference ***0018.

The proof

The receipt states the delivered bytes.

The runner reads the object back from S3: same size, same SHA-256.

Replay of 3 November 2026.

The limit

Masking is not encryption.

  • A masked identifier can still be correlated.
  • If the partner must not correlate, drop the column.
  • Snapshot mode rewrites the whole object.

Try it

The scenario replays end to end.

CSV, entity, S3 export: every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →