lumnik. / scenarios All scenarios · Français

One unexpected value in 500 as a dead letter and the other 499 written

Goal Keep one unexpected line from stopping the other 499, and keep it visible.

A goods receipt typed by hand: 500 lines, quantities like « 12,5 kg » or « 3 pce ». An operator's Kotlin script reads them. Line 317 says « env. 4 kg ».

manifests/quantity-v1.kts
// Operator-provided: reads a quantity written "12,5 kg" or "3 pce" into a number and a unit.@Suppress("UNCHECKED_CAST")val cells = bindings["cells"] as MutableMap<String, Any?>val (amount, unit) = (cells["quantity"] as String).trim().split(" ")cells["qty"] = amount.replace(',', '.').toBigDecimal().toPlainString()cells["unit"] = unit
receipts.csv
line_no;item;quantity
L0001;Sel fin;21 pce
L0002;Farine T55;4 pce
L0003;Levure fraîche;18 pce
L0004;Œufs plein air;18,5 kg
L0005;Farine T55;4,0 kg
L0006;Sel fin;18,4 kg
L0007;Beurre doux;29,7 kg
L0008;Farine T55;32,0 kg
L0009;Sucre roulé;17,7 kg
L0010;Levure fraîche;20,1 kg
L0011;Sel fin;15,1 kg
L0012;Beurre doux;37,2 kg
L0013;Œufs plein air;36,7 kg
L0014;Farine T55;37,9 kg
L0015;Levure fraîche;28,7 kg
L0016;Sucre roulé;28,2 kg
L0017;Sucre roulé;15,6 kg
L0018;Œufs plein air;8 pce
L0019;Levure fraîche;13,1 kg

… 481 more lines

manifests/quantity-v2.kts
// Operator-provided: reads a quantity written "12,5 kg" or "3 pce" into a number and a unit.// A quantity in any other shape is declared unreadable: counted, never guessed.@Suppress("UNCHECKED_CAST")val cells = bindings["cells"] as MutableMap<String, Any?>val m = Regex("(\\d+(?:,\\d+)?) (kg|pce)").matchEntire((cells["quantity"] as String).trim())if (m == null) {    (bindings["unreadable"] as Runnable).run()} else {    cells["qty"] = m.groupValues[1].replace(',', '.')    cells["unit"] = m.groupValues[2]}
manifests/11-receipts-dlq.yaml
apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: scenario-receipts-b  connector_type: csv-file  scopes: [warehouse-b]spec:  transport:    kind: local    path: /tmp/lumnik-scenario/quantity-script-one-in-500/b/receipts.csv  parser:    delimiter: ";"  after_process: none  mapping:    on-error:      default: dlq                     # a line the script declares unreadable goes to the dead letters, as read    cell:      - kotlin:          script: hooks:quantity-b.kts   # the operator's reading of "12,5 kg" / "3 pce"
manifests/20-receipt-line.yaml
kind: Entityname: receipt_line_blabel: Ligne de réceptiondescription: "A received line: the item and its quantity read as a number and a unit."match: [line_no]scopes: [warehouse-b]columns:  line_no: { label: "Ligne" }  item: { label: "Article" }  quantity: { label: "Quantité saisie" }  qty: { label: "Quantité" }  unit: { label: "Unité" }sources:  - table: t_scenario_receipts_b    map: { line_no: line_no, item: item, quantity: quantity, qty: qty, unit: unit }
$ lm dlq list --run 897ccf78-90fa-4710-b4a8-88f74d003bfd
ID        CONNECTOR            RUN       REASON         ATTEMPTS  RESOLUTION  MESSAGE
3a59d4c6  scenario-receipts-b  897ccf78  MappingFailed  1                     kotlin: kotlin could not read the value

$ lm dlq get 3a59d4c6-3d6c-4af2-ab26-3c0d91408113
ID:        3a59d4c6-3d6c-4af2-ab26-3c0d91408113
Connector: scenario-receipts-b (4bd712c0-288b-413e-8ad3-60661f10381d)
Run:       897ccf78-90fa-4710-b4a8-88f74d003bfd
Reason:    MappingFailed
Attempts:  1
Message:   kotlin: kotlin could not read the value
Detail:    {"field":"kotlin","message":"kotlin could not read the value"}
Payload:   {"item":"Sel fin","line_no":"L0317","quantity":"env. 4 kg"}
Resolution: open

Limits

  • A failing script's message gives the script's line, not the file's: a script that declares the value unreadable is what tells which line is the problem.
  • The dead letter names the kotlin hook, not a column: a script can write several cells.
  • Synthetic lines.

Receiving · typed in

A goods receipt typed by hand.

500 lines, quantities as text.

12,5 kg, 3 pce… and env. 4 kg

The problem

A value nobody expected.

  • « env. 4 kg », line 317.
  • The script cannot read it.

What is not changed

The entry stays as it is.

  • The fix is made at the source.
  • The hub does not guess.

The trap

One line stops everything.

  • The script throws.
  • None of the 500 lines is written.

What had to be written

on-error: { default: dlq }

One line.

  • The script says unreadable instead of guessing.
  • The line becomes a dead letter.
  • A 34-line scenario, two manifests.

The result

499 written, one set aside.

  • L0317, as it was read.
  • L0316 and L0318 in the table.

The limit

An exception gives the script's line.

  • Not the file's.

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →