+ jHj ~Rt^RIHt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI H t H t H t Ht^RIHt^RIHt^RIHt]P*!]4t]P0!4t^tRPtRQtRtRtR tR t ]!!R 04t"RRt#]$!]#4]^, 8XgQhR t%R t&RSt'Rt(]PR!R]PT4t+RRlt,RRlt-RRlt.]&3RRllt/RRlt0RRlt1RRlt2RRlt3R t4R!R"lt5R#R$lt6R%t7R&R'lt8R(R)lt9R*R+lt:R,R-lt;R.R/R/R0lltR5R6lt?RTR7R8llt@RTR9R:lltARTR;R<lltBR=R>R?R>R@R/RARBlltCRURCRRDR/RERFllltDRURGR/RHRIllltERURJRKlltFRLRMltGRURNROlltHR#)VaDurable delivery-obligation ledger for gateway final responses (rows in the shared ``state.db``; WAL, owner pid + process-start liveness, bounded retention) so a crash between finalize and platform ACK cannot lose a response silently. Checkpoints: record_obligation() 'pending' before any send | mark_attempting() 'attempting' right before the await | mark_delivered() 'delivered' only on SendResult.success | mark_failed() 'failed' on a definitive rejection. Crash semantics (never silently resend an ambiguous send): pending = never started, redeliver plainly; attempting = crashed mid-await, platform MAY have it, redeliver WITH a visible recovered marker; failed = rejected once, restart is a retry boundary, also marked; delivered = prune. Attempts are capped and stale rows expire, both -> 'abandoned' (kept briefly, then pruned). Everything is best-effort: ledger failures must never block a send; callers wrap every call in try/except. ) annotationsN)AnyDictListOptional)classify_dead_error)add_column_if_missingget_process_hermes_homeiu_♻️ Recovered reply — the gateway restarted during delivery, so this may be a duplicate: u♻️ Recovered reply — the messaging platform reconnected after the original delivery failed, so this may be a duplicate: u♻️ Recovered reply — the messaging platform's rate limit refused the original, so part of it may already have arrived above: send_path_degradedzflood_control:gN@g@z3flood control exceeded.*?retry in\s+(\d+(?:\.\d+)?)c V^8dQhRRRR/#)textstrreturnOptional[float])formats"`` result, or a row still carrying the platform's own flood wording (see ``_RAW_FLOOD_RE``).rN)rr&r' startswithFLOOD_ERROR_PREFIXr )r#rs& ris_flood_errorr/esH u{{  ! ! # ) ) +D ??- . S S/$2Gt2SSrc$V^8dQhRRRRRR/#)r r#rdefaultrrr)rs"rrrls"))c)E)TY)rcR\T;'gR4P4P4pTpVP\4'd,\ V\ \4RP44pM\V4pVeTpV^8dV#T# \dTpLi;i)zdThe wait the platform asked for, read from ``flood_control:``; ``default`` when unreadable.rN) rr&r'r-r.rlenrr )r#r1rwaitraws&& rflood_wait_secondsr6ls u{{  ! ! # ) ) +D D )** c"4567==?@D d# ?D!84(( D s)B B&%B&c V^8dQhRRRR/#)r secondsrrrr)rs"rrr~sTTsTuTrc\V4p\ \ VR4\ 4\,# \\3d \pL>i;i)aHow long a redelivery timer sleeps for a wait of ``seconds``: capped so a huge or bogus value cannot park the timer for hours (the row's own deadline, not the timer, decides eligibility when it fires), plus a little slack so the timer lands after the deadline rather than on it.)rrrFLOOD_RETRY_DEFAULT_SECONDSminmaxFLOOD_RETRY_CAP_SECONDSFLOOD_RETRY_SLACK_SECONDS)r8r4s& rflood_retry_delayr@~sG+W~ s4~6 7:S SS z "+*+s 2A  A c V^8dQhRRRR/#)r updated_atrrrr)rs"rrrscerc^\T;'gR4# \\3dR#i;i)r:)rrr)rBs&r _failed_stamprDs1Z&&3'' z "s ,,c$V^8dQhRRRRRR/#)r rBr last_errorrrr)rs"rrrs&FFF#F%Frc8\V4\V4,#)aEarliest moment a flood-refused row may be resent: the refusal's timestamp (``mark_failed`` sets ``updated_at``) plus the platform's wait. Enforced by the sweeps so neither an early timer nor a reconnect sweep spends a redelivery attempt inside the penalty window.)rDr6)rBrFs&&rflood_not_beforerHs  $'9*'E EErc(V^8dQhRRRRRRRR/#)r rBrrFattemptsrrr)rs"rrrs.EEE#EEErc\V4'd \W4#\T;'gR4P4P 4p\ V4'd \ V4#\V4'dR#\T;'g^4pV\^, 8dR#\ V4\V,,#)uEarliest moment a failed row may be resent, or ``None`` for a row the runtime must leave alone: a flood refusal keeps the platform's own wait, an allowlisted reconnect error is due at once, a whole-chat death is final, and any other rejection backs off by the attempts already spent — but never spends the LAST budgeted attempt. An unclassified outage can outlast any timer, and a row the timer abandoned would be lost for good; leaving one attempt keeps it recoverable by the boot sweep after a restart, which is a real recovery signal.rN) r/rHrr&r'r*rDrint MAX_ATTEMPTS_RETRY_BACKOFF_SECONDS)rBrFrJrspents&&& rretry_not_beforerPsj!! 77 zR & & ( . . 0DZ((4    A E q    $'=e'D DDrc$\4R, #)zstate.dbr rrr_db_pathrRs # $z 11rcV^8dQhRR/#)r rsqlite3.Connectionr)rs"rrrsDD$Drc@^RIHpV!\4RRR\R7#)open_dbzstate.db (delivery_ledger)i'N)db_labelbusy_timeout_ms row_factory initialize)hermes_cli.sqlite_utilrXrR_initialize_schemarWs r_connectr_s&. 8:(DV\#0B DDrc V^8dQhRRRR/#)r connrTrNoner)rs"rrrsgg/gDgrcVPR4RVPR4Uu0uF q^,kK up9d\VRRR4R#R#uupi)a5CREATE TABLE IF NOT EXISTS delivery_obligations ( obligation_id TEXT PRIMARY KEY, session_key TEXT NOT NULL, platform TEXT NOT NULL, chat_id TEXT NOT NULL, thread_id TEXT, content TEXT NOT NULL, state TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0, created_at REAL NOT NULL, updated_at REAL NOT NULL, owner_pid INTEGER, owner_started_at INTEGER, last_error TEXT, adapter_profile TEXT )adapter_profilez'PRAGMA table_info(delivery_obligations)delivery_obligationszadapter_profile TEXTN)executer)rarows& rr^r^sVLL $4<<@i3j k3jCQ3j kkd$:>[M;-q BII'S\] ^ h h jkn oorrdc8V^8dQhRRRRRRRRRRRRR RR R /#) r obligation_idrrplatformchat_id thread_id Optional[str]rrdrrbr)rs"rrrsI######WZ#!.#9<#O\#hl#rc\P!4\4upwr\;_uu_4\4;_uu_4p T P RYT\ V4V'd \ V4MRYWYxY'd\ V4P 4MR3 4\W4RRR4RRR4R# +'giL;i +'giR#;i)zBRecord a final response as owed to the platform (state='pending').a;INSERT OR REPLACE INTO delivery_obligations (obligation_id, session_key, platform, chat_id, thread_id, content, state, attempts, created_at, updated_at, owner_pid, owner_started_at, adapter_profile) VALUES (?, ?, ?, ?, ?, ?, 'pending', 0, ?, ?, ?, ?, ?)Nr1)timerx_DB_LOCKrkrfrr&_prune_unlocked) rrrrrrrdnowrmstartedras $$$$$$$ rrecord_obligationrs))+|~C# <>>T  I 3w>s$C A"B8 &C 8 C C  C c<V^8dQhRRRRRRRRRRRRR R R RR R / #)r rrrrrrrrsincerrdrrbr)rs"rrr sQ++c++s+]`+'4+?B+KP+-:+FJ+rc\P!4p\;_uu_4\4;_uu_4p T PRYT\ V4V'd \ V4MRYXY'd\ V4P 4MRWV3 4RRR4RRR4R# +'giL;i +'giR#;i)agAdopt a reply a killed process persisted but never ledgered. Unowned, so this boot's sweep claims it, and 'attempting', because a streamed reply may already be on screen: it is redelivered once, with the recovered marker. A no-op when the same reply was already ledgered since *since* (the turn start), and idempotent across boots that die before their sweep.aINSERT OR IGNORE INTO delivery_obligations (obligation_id, session_key, platform, chat_id, thread_id, content, state, attempts, created_at, updated_at, owner_pid, owner_started_at, adapter_profile) SELECT ?, ?, ?, ?, ?, ?, 'attempting', 0, ?, ?, NULL, NULL, ? WHERE NOT EXISTS (SELECT 1 FROM delivery_obligations WHERE session_key = ? AND content = ? AND created_at >= ?)Nr1)rrrkrfrr&) rrrrrrrrdrras $$$$$$$$ rrecord_crash_left_replyr s ))+C <>>T  _3w>s#B4AB! B4! B1 ,B44 C c V^8dQhRRRR/#r rrrrbr)rs"rrr6s//3/4/rc\VR4R#) attemptingN _update_staters&rmark_attemptingr6s -.rc V^8dQhRRRR/#rr)rs"rrr:s..#.$.rc\VR4R#) deliveredNrrs&rmark_deliveredr:s --rc$V^8dQhRRRRRR/#)r rrr#rrbr)rs"rrr>s!88s8388rc"\VRVR7R#)failedr)Nr)rr#s&&r mark_failedr>s-7rc$V^8dQhRRRRRR/#)r rrr#rr$r)rs"rrrBs!!!!S!$!rc t\4wr#VfR#\;_uu_4\4;_uu_4pTPR\P!4V'd VR,MRWV34pRRR4RRR4\ XP 4# +'giL-;i +'giL8;i)alReturn an unsent runtime claim to ``failed`` without spending an attempt. Runtime recovery claims before clearing ``resume_pending`` so two reconnect paths cannot send the same row; if the flag cannot be cleared no send was attempted and the claim must not consume the redelivery budget. Fail-closed to the exact current process instance and ``attempting`` state.NFa;UPDATE delivery_obligations SET state='failed', attempts=CASE WHEN attempts > 0 THEN attempts - 1 ELSE 0 END, updated_at=?, last_error=? WHERE obligation_id=? AND state='attempting' AND owner_pid IS ? AND owner_started_at IS ?NiN)rxrrkrfrr$rowcount)rr#rmrracursors&& rrelease_runtime_claimrBsx  >LC <>>T A YY[%+D-g VX"   ">s"B';B /B' B$ B'' B7 c(V^8dQhRRRRRRRR/#)r rrstater#rrbr)rs"rrrWs.QQQSQQdQrc ,\;_uu_4\4;_uu_4pTPRT\P!4V'd VR,MRV34RRR4RRR4R# +'giL;i +'giR#;i)zwUPDATE delivery_obligations SET state=?, updated_at=?, last_error=? WHERE obligation_id=?rN)rrkrfr)rrr#ras&&& rrrWsR <>>T  (DIIKt4 O  Q">>s"B;A/ B/ A? :B B runtimeFfloodrFc ,V^8dQhRRRRRRRRRR/#) r needs_markerr$rrrFrrzdict[str, Any]r)rs"rrr`s; \ \# \.2 \CG \* \6D \rcV 'd\MV 'd\MRp RVRVRVRVRVRVRV/V'd V 'dR V /M/CR V/CV 'dR R /M/CV 'dR V /M/CRV^,/C#)aClaimed-row dict handed back for redelivery. A marked row names its own cause: ``flood`` (a reply the rate limit refused, possibly after accepting part of it) gets FLOOD_MARKER at boot or at runtime, a ``runtime`` reconnect replay gets RECONNECTED_MARKER, and a boot-recovered crash keeps the runner's restart marker default. ``last_error`` is the row's pre-claim error, carried so a runtime claim that is released unsent goes back to ``failed`` with the same error and keeps its retry eligibility.Nrrrrrrrmarkerprofileruntime_recoveryTrFrJ) FLOOD_MARKERRECONNECTED_MARKER) oidrrrrrrJrrrrrFrs &&&&&&&&$$$$ r _claimed_rowr`s#\w);DF S-j(T]_f Iw  \%1f&!" \GPRY \.5"D)" \.8 j)R  \CMhYZl  \\rdeliverable_platformsdeliverable_targetsc(V^8dQhRRRRRRRR/#)r rrrz Optional[set]rrlist[dict[str, Any]]r)rs"rrrps0NN?N]N+8NDXNrcBVeTM\P!4\4upwr4.p\;_uu_4\4;_uu_4pVP R4P 4pVEFuwrrrrpppppp\ VV4'dK#V\8gVV, \8dVP RW34KUVeW9gVe V V3V9dKnVR8H;'d \V4pV'dvV\VV48deVP RW4VVV34pVP'd:TPRTRT RT R T R T R T R T;'gR RVRRR\VV4/ 4EKTP RY4TV'd^M^VVV34pVP'gEK:TP\YYYTT;'gR VR8g;'gV^8VR7 4EKx RRR4RRR4V# +'giL;i +'giT#;i)aClaim undelivered rows owned by dead processes; return them for redelivery. Claiming atomically re-stamps the owner to THIS process, moves the row to 'attempting' and increments ``attempts`` (the UPDATE is guarded on the previous owner stamp, so a second gateway racing the same sweep cannot double-claim). Rows over the attempts cap or stale cutoff become 'abandoned'. ``deliverable_platforms`` restricts claiming to platforms the caller can send on this boot: ``attempts`` is the redelivery budget and must only be spent on a real send, else a platform that failed to connect burns one attempt per boot and hits the cap having never been sent once (the stale cutoff still bounds untouched rows). ``deliverable_targets`` further scopes multiplexed gateways by exact ``(platform, adapter_profile)`` so one connected bot cannot spend another disconnected bot's retry budget. A flood-refused row still inside its wait is adopted (owner re-stamped, no attempt spent) and returned flagged ``adopted`` with its ``not_before``: the caller clears its session's resume flag like any other claimed row, since the answer is in the ledger, but must not send it; the flood timer does once the wait has passed. A legacy row without ``adapter_profile`` is normalised to ``'default'`` on claim or adoption (the caller only accepts such rows when it is not multiplexed), because the runtime sweep matches profiles exactly and could otherwise never claim it.NaBSELECT obligation_id, session_key, platform, chat_id, thread_id, content, state, attempts, created_at, owner_pid, owner_started_at, adapter_profile, last_error, updated_at FROM delivery_obligations WHERE state IN ('pending', 'attempting', 'failed')zlUPDATE delivery_obligations SET state='abandoned', updated_at=? WHERE obligation_id=?rzUPDATE delivery_obligations SET owner_pid=?, owner_started_at=?, adapter_profile=COALESCE(adapter_profile, 'default') WHERE obligation_id=? AND (owner_pid IS ? OR owner_pid=?)rrrrrrrr1rJadoptedT not_beforeapUPDATE delivery_obligations SET owner_pid=?, owner_started_at=?, attempts=attempts+1, updated_at=?, adapter_profile=COALESCE(adapter_profile, 'default'), state='attempting', last_error=CASE WHEN ? THEN NULL ELSE last_error END WHERE obligation_id=? AND (owner_pid IS ? OR owner_pid=?)pending)rr)rrxrrkrffetchallrrMSTALE_AFTER_SECONDSr/rHrappendr)rrrrmrclaimedrarowsrrrrrrrrJ created_at owner_pidowner_started_atrdrFrB flood_rowrs&$$ rsweep_recoverablerps2("%#diik<>C#$&G <>>T|| E (* W[ SSx)eWa (/:zI'788<'C*,<@S+S TVYU_a&2x7\+7X)9K9[9[xZ[|cl noYW["p Nq">p NsDH  BG: (G: C# (/7i3GySQX\$&G <>>T|| 69A{ D EMHJ W[ SS|i(T^ (*oz"22i36FJZ^eJe":z8DC{Y0@AK<'C*,<@S+S IKV W Sy\\EGR SF |ClU^,4oTXbf2@2LYc ef9W["N NO">N Ns+F.BC# <>>T|| TVYUcefnemeo " $&HSWO:8 Z ( ( zx@ ;(l2s7GK^6^ 55I6  # "5SMTX-38>>3C,D F,D(#S9g|S I,D FF#">" Fs)E"D0 2EE0 E ;E E c V^8dQhRRRR/#)r rrrrbr)rs"rrrs44u44rcVPRV\, 34VPR4P4^,pV\8d!VPRV\, 34R#R#)u_Retention DELETEs on the caller's open connection — must run inside the caller's transaction.zhDELETE FROM delivery_obligations WHERE state IN ('delivered', 'abandoned') AND updated_at < ?z)SELECT COUNT(*) FROM delivery_obligationsaqDELETE FROM delivery_obligations WHERE obligation_id IN ( SELECT obligation_id FROM delivery_obligations ORDER BY CASE state WHEN 'delivered' THEN 0 WHEN 'abandoned' THEN 1 ELSE 2 END, updated_at ASC LIMIT ?)N)rf_RETENTION_SECONDSfetchone _MAX_ROWS)rartotals&& rrrslLL KMPSeMeLgi LLD E N N PQR SE y   %y02 4rc V^8dQhRRRR/#)r configzOptional[dict[str, Any]]rr$r)rs"rrr+s  3 t rc$Vf^RIHpV!4pVPR4;'g/PRR4p\V\4'd"VP 4P 4R9#\V4# \dR#i;i)z>Read the ``gateway.delivery_ledger`` config gate (default on).) load_configgatewaydelivery_ledgerT>0noofffalse) hermes_cli.configrgetrrr&r'r$rq)rrvalues& rledger_enabledr+s > 5 ]FI&,,"112CTJKUV[]`KaKau{{}""$,GGrgklqgrr s(BA B5 B BBiQi: )g>@g^@g @)rru)I__doc__ __future__rrloggingrvresqlite3 threadingrtypingrrrrgateway.dead_targetsrr]rhermes_constantsr getLogger__name__loggerLockrrMrrrRECOVERED_MARKERrr frozensetr(rNr3r.r;r>r?compile IGNORECASErr r*r/r6r@rDrHrPrRr_r^rkrrrxrrrrrrrrrrrrrrrrrrrs #  ,,484   8 $ >>  "%  wI9 &';&<=& !"lQ&6666&"# QSUS`S`a I T5P)$TFE(2Dg.# ! Dp #_c#$+=A+,/.8!*Q \5: \JO \-1 \ N]aN;?NNb86:88vF:4$  r