lumnik. / scenarios All scenarios

The SMTP relay changed its password: three cold-room alerts do not leave, the on-call desk knows

Goal Mail a cold room's temperature excursions, know the same day that a mail did not leave and why, then send it again without a duplicate.

A pharmaceutical wholesaler stores its medicines between 2 and 8 °C in two cold rooms. The data logger exports its readings once a day; an excursion is mailed to the quality lead, through the company's SMTP relay.

cold-rooms-20261108.csv
reading_id;room;probe;read_at;temp_c
CF-1-2026110800;CF-1;PT100-0412;2026-11-08 00:00;4.7
CF-2-2026110800;CF-2;PT100-0418;2026-11-08 00:00;6.1
CF-1-2026110801;CF-1;PT100-0412;2026-11-08 01:00;5.3
CF-2-2026110801;CF-2;PT100-0418;2026-11-08 01:00;5.2
CF-1-2026110802;CF-1;PT100-0412;2026-11-08 02:00;6.0
CF-2-2026110802;CF-2;PT100-0418;2026-11-08 02:00;5.8
CF-1-2026110803;CF-1;PT100-0412;2026-11-08 03:00;5.2
CF-2-2026110803;CF-2;PT100-0418;2026-11-08 03:00;5.4
CF-1-2026110804;CF-1;PT100-0412;2026-11-08 04:00;4.0
CF-2-2026110804;CF-2;PT100-0418;2026-11-08 04:00;6.2
CF-1-2026110805;CF-1;PT100-0412;2026-11-08 05:00;5.8
CF-2-2026110805;CF-2;PT100-0418;2026-11-08 05:00;4.0
CF-1-2026110806;CF-1;PT100-0412;2026-11-08 06:00;6.0
CF-2-2026110806;CF-2;PT100-0418;2026-11-08 06:00;4.8
CF-1-2026110807;CF-1;PT100-0412;2026-11-08 07:00;4.9
CF-2-2026110807;CF-2;PT100-0418;2026-11-08 07:00;4.8
CF-1-2026110808;CF-1;PT100-0412;2026-11-08 08:00;5.5
CF-2-2026110808;CF-2;PT100-0418;2026-11-08 08:00;4.6
CF-1-2026110809;CF-1;PT100-0412;2026-11-08 09:00;4.9

… 29 more lines

cold-rooms-20261109.csv
reading_id;room;probe;read_at;temp_c
CF-1-2026110900;CF-1;PT100-0412;2026-11-09 00:00;3.9
CF-2-2026110900;CF-2;PT100-0418;2026-11-09 00:00;4.8
CF-1-2026110901;CF-1;PT100-0412;2026-11-09 01:00;4.2
CF-2-2026110901;CF-2;PT100-0418;2026-11-09 01:00;4.4
CF-1-2026110902;CF-1;PT100-0412;2026-11-09 02:00;4.2
CF-2-2026110902;CF-2;PT100-0418;2026-11-09 02:00;3.8
CF-1-2026110903;CF-1;PT100-0412;2026-11-09 03:00;4.5
CF-2-2026110903;CF-2;PT100-0418;2026-11-09 03:00;4.7
CF-1-2026110904;CF-1;PT100-0412;2026-11-09 04:00;5.2
CF-2-2026110904;CF-2;PT100-0418;2026-11-09 04:00;6.1
CF-1-2026110905;CF-1;PT100-0412;2026-11-09 05:00;5.2
CF-2-2026110905;CF-2;PT100-0418;2026-11-09 05:00;4.1
CF-1-2026110906;CF-1;PT100-0412;2026-11-09 06:00;4.2
CF-2-2026110906;CF-2;PT100-0418;2026-11-09 06:00;4.5
CF-1-2026110907;CF-1;PT100-0412;2026-11-09 07:00;4.0
CF-2-2026110907;CF-2;PT100-0418;2026-11-09 07:00;4.5
CF-1-2026110908;CF-1;PT100-0412;2026-11-09 08:00;4.2
CF-2-2026110908;CF-2;PT100-0418;2026-11-09 08:00;5.5
CF-1-2026110909;CF-1;PT100-0412;2026-11-09 09:00;4.3

… 29 more lines

IT
$ relay: alertes-qualite now logs in with the new password only

evidence: replay.log, hub-monday.txt, deliveries-monday.txt, relay-inbox-monday.txt, relay-tuesday.txt, inbox-tuesday.txt, deliveries-tuesday.txt, webhook-received.txt, hub-fixed.txt, replay.txt, replay-sent.txt, deliveries-fixed.txt, relay-inbox-fixed.txt · transform v1

cold-rooms-20261110.csv
reading_id;room;probe;read_at;temp_c
CF-1-2026111000;CF-1;PT100-0412;2026-11-10 00:00;4.1
CF-2-2026111000;CF-2;PT100-0418;2026-11-10 00:00;5.1
CF-1-2026111001;CF-1;PT100-0412;2026-11-10 01:00;6.2
CF-2-2026111001;CF-2;PT100-0418;2026-11-10 01:00;6.0
CF-1-2026111002;CF-1;PT100-0412;2026-11-10 02:00;8.4
CF-2-2026111002;CF-2;PT100-0418;2026-11-10 02:00;5.1
CF-1-2026111003;CF-1;PT100-0412;2026-11-10 03:00;9.7
CF-2-2026111003;CF-2;PT100-0418;2026-11-10 03:00;4.5
CF-1-2026111004;CF-1;PT100-0412;2026-11-10 04:00;10.3
CF-2-2026111004;CF-2;PT100-0418;2026-11-10 04:00;3.8
CF-1-2026111005;CF-1;PT100-0412;2026-11-10 05:00;3.9
CF-2-2026111005;CF-2;PT100-0418;2026-11-10 05:00;6.0
CF-1-2026111006;CF-1;PT100-0412;2026-11-10 06:00;4.5
CF-2-2026111006;CF-2;PT100-0418;2026-11-10 06:00;3.9
CF-1-2026111007;CF-1;PT100-0412;2026-11-10 07:00;5.8
CF-2-2026111007;CF-2;PT100-0418;2026-11-10 07:00;3.9
CF-1-2026111008;CF-1;PT100-0412;2026-11-10 08:00;4.2
CF-2-2026111008;CF-2;PT100-0418;2026-11-10 08:00;4.9
CF-1-2026111009;CF-1;PT100-0412;2026-11-10 09:00;4.5

… 29 more lines

$ lm notify replay 17
delivery 17: sent

$ lm notify replay 18
delivery 18: sent

$ lm notify replay 19
delivery 19: sent

$ lm notify replay 16
Error: API error 409: this mail was sent; it is not sent twice

$ lm notify deliveries
ID  RECIPIENT                        EVENT                            STATUS          ATTEMPTS  WHEN              ERROR
11  qualite-m2@pharma-depot.example  decision.cold-room-excursion-m2  given up: send  3         2026-11-08 21:18  authentication refused (SMTP 535)
10  qualite-m2@pharma-depot.example  decision.cold-room-excursion-m2  given up: send  3         2026-11-08 21:18  authentication refused (SMTP 535)
9   qualite-m2@pharma-depot.example  decision.cold-room-excursion-m2  given up: send  3         2026-11-08 21:18  authentication refused (SMTP 535)
7   qualite-m1@pharma-depot.example  decision.cold-room-excursion-m1  given up: send  3         2026-11-08 21:13  authentication refused (SMTP 535)
6   qualite-m1@pharma-depot.example  decision.cold-room-excursion-m1  given up: send  3         2026-11-08 21:13  authentication refused (SMTP 535)
5   qualite-m1@pharma-depot.example  decision.cold-room-excursion-m1  given up: send  3         2026-11-08 21:13  authentication refused (SMTP 535)
19  qualite-a@pharma-depot.example   decision.cold-room-excursion-a   sent            4         2026-11-08 21:56  
18  qualite-a@pharma-depot.example   decision.cold-room-excursion-a   sent            4         2026-11-08 21:56  
17  qualite-a@pharma-depot.example   decision.cold-room-excursion-a   sent            4         2026-11-08 21:56  
16  qualite-a@pharma-depot.example   decision.cold-room-excursion-a   sent            1         2026-11-08 21:55  
15  qualite-m3@pharma-depot.example  decision.cold-room-excursion-m3  sent            4         2026-11-08 21:20  
14  qualite-m3@pharma-depot.example  decision.cold-room-excursion-m3  sent            4         2026-11-08 21:20  
13  qualite-m3@pharma-depot.example  decision.cold-room-excursion-m3  sent            4         2026-11-08 21:20  
12  qualite-m3@pharma-depot.example  decision.cold-room-excursion-m3  sent            1         2026-11-08 21:19  
8   qualite-m2@pharma-depot.example  decision.cold-room-excursion-m2  sent            1         2026-11-08 21:17  
4   qualite-m1@pharma-depot.example  decision.cold-room-excursion-m1  sent            1         2026-11-08 21:12  
3   qualite@example.com              decision.trial-mail-excursion    sent            4         2026-11-08 17:15  
2   qualite@example.com              decision.trial-mail-excursion    sent            4         2026-11-08 17:15  
1   qualite@example.com              decision.trial-mail-excursion    sent            1         2026-11-08 17:08

Limits

  • The replayed mail arrives after the outage: here, hours after the excursion. The hub does not block the lots; it tells.
  • A changed password gives up every mail: one event per mail, not one per outage.
  • The webhook carries the mail's ID and the cause, never the alert's text: the on-call desk knows a mail must be sent again, not what it said.
  • Mails are sent again one by one.
  • Passwords change in the hub's configuration, with a restart.
  • Readings, rooms and probes synthetic; the relay is a local Mailpit.

Cold rooms · 2-8 °C

An excursion is mailed.

To the quality lead, through the company's SMTP relay.

temp_c>8

The problem

The relay changes its password. The hub keeps the old one.

  • Three alerts of CF-1 do not leave.
  • The quality lead receives nothing, and does not know there was something to receive.

What is not changed

The relay and the data logger stay as they are.

  • The export is the data logger's own.
  • The hub logs in to the relay like any other software.

What had to be written

lm integration add relay-watch \
  --events notification.mail.given-up

A given-up mail tells the on-call desk.

  • Then the new password.
  • And lm notify replay for each mail.

What was fixed

A refused password and a dead relay read the same (#1905).

  • A given-up mail is an event (#1906).
  • The mail's subject is the decision's sentence (#1910).

Try it

The scenario replays locally.

Every step is published with its output.

This scenario runs on lumnik Open, free and open source.

Try lumnik Open →