+ jHj0a0t$Rt^RIHt^RIt^RIt^RIt^RIt^RIH t ^RI H t ^RI H t HtHtHtHt^RIHt^RIHt^RIHtRtR ]R &R2tR3tR t^t]P@!R 4t!]PD!4t#R Rlt$RRlt%RRlt&] RRl4t'RRlt(RRlt)RRlt*RRlt+RRlt,RRR RR!R/R"R#llt-R$R%lt.R&R'lt/R(R)lt0R*R+lt1R4R,R-llt2R.R/lt3R4R0R1llt4R#)5uDurable cron failure incidents with signature dedup and ack. The executions ledger records every attempt; this module groups the *failures* into incidents keyed by ``(job_id, error signature)`` so the same job failing with the same error does not re-ping the operator every run once acknowledged. Lifecycle: ``detected`` → ``alerted`` → ``closed``. The same job + same normalized error resolves to the SAME incident id, so a closed incident stays closed until the error text changes and mints a new one. ``alerted`` means a failure ping actually reached the operator (``alerted_at`` = when the latest one did; the scheduler withholds repeats until ``cron.failure_repeat_alert_hours`` have passed). Incidents share ``cron/executions.db`` with ``cron.executions`` (one ledger file). ) annotationsN)contextmanager)Path)AnyDictIteratorListOptional) executions)get_hermes_home)nowzOptional[Path]EXECUTIONS_FILEizJ\b\d+(?:\.\d+)?\s*(?:ms|s|secs?|seconds?|m|mins?|minutes?|h|hrs?|hours?)\bcV^8dQhRR/#)returnr)formats"3/home/ubuntu/.hermes/hermes-agent/cron/incidents.py __annotate__r5sBB$Bc\P\3FpVfK \V4u# \4P 4R, R, #)zShared cron DB path. The ``cron.executions`` override wins when installed so redirecting the executions ledger also redirects the incident table (they must stay in the SAME database); then this module's own override, then the canonical profile home.cronz executions.db) _executionsr rr resolve)overrides r_db_pathr5sH!00/B  > !C   $ $ & // AArcV^8dQhRR/#)rrsqlite3.Connectionr)rs"rrr?s n n$ nrcr^RIHp^RIHp\ 4pV!VP 4V!VRR\ R7#))_ensure_cron_dir)open_dbzcron/executions.dbT)db_labelsynchronous_full initialize) cron.jobsr hermes_cli.sqlite_utilr!rparent_initialize_schema)r r!paths r_connectr*?s0+. :DT[[! 4"6Zl mmrc V^8dQhRRRR/#)rconnrrNoner)rs"rrrKs/Drc^RIHpVPR4V!VRRR4VPR4VPR4R#) r)add_column_if_missinga CREATE TABLE IF NOT EXISTS cron_incidents ( id TEXT PRIMARY KEY, job_id TEXT NOT NULL, error_sig TEXT NOT NULL, state TEXT NOT NULL, failure_type TEXT NOT NULL DEFAULT 'unknown', first_seen_at TEXT NOT NULL, last_seen_at TEXT NOT NULL, acked_at TEXT, alerted_at TEXT, closed_at TEXT, error TEXT NOT NULL, output_file TEXT )cron_incidents alerted_atzalerted_at TEXTzKCREATE INDEX IF NOT EXISTS idx_cron_incidents_job ON cron_incidents(job_id)zLCREATE INDEX IF NOT EXISTS idx_cron_incidents_state ON cron_incidents(state)N)r&r/execute)r,r/s& rr(r(KsL<LL "$ 0,@QRLL $ LL #rcV^8dQhRR/#)rrzIterator[sqlite3.Connection]r)rs"rrrks2rc #"^RIHp\;_uu_4V!\44;_uu_4pVxRRR4RRR4R# +'giL;i +'giR#;i5i)r) transactionN)r&r5_lockr*)r5r,s r _transactionr7js62  HJ''4 (''s1A.AA A A. A A A+ % A.c V^8dQhRRRR/#rerrorstrrr)rs"rrrrsAACACArc\P!RR\T;'gR44P4P 4#)zDStrip whitespace and lowercase before signing (dedup normalization).z\s+ )resubr;striplower)r:s&r_normalize_errorrCrs2 66&#s5;;B/ 0 6 6 8 > > @@rc V^8dQhRRRR/#r9r)rs"rrrws " " " "rc\T;'gR4p^RIHpV!VRR7pVR\# \dLi;i)zYRedact secrets (best-effort; the scheduler path never fails on it) then bound the length.r>)redact_sensitive_textT)forceN)r; agent.redactrF ExceptionMAX_ERROR_CHARS)r:textrFs& r _redact_errorrLwsJ u{{ D 6$T6   !!    s 0 >>c$V^8dQhRRRRRR/#)rjob_idr;r:rr)rs"rrrs&RRSRRRrc\PR\V44R\p\P !VP 4VP 4,4P4R,#)z>Dedup key: stable for same job + same normalized error prefix.z#sN:N N) _DURATION_REr@rC_MAX_SIGNATURE_ERROR_CHARShashlibsha256encode hexdigest)rNr: normalizeds&& r_error_signaturerXsT!!$(8(?@A\B\]J >>&--/J,=,=,?? @ J J LS QQrc$V^8dQhRRRRRR/#)rrNr; error_sigrr)rs"rrrs!'''''rcVR, RV 2#):NN_r)rNrZs&&r _incident_idr^sRj\9+ &&rc V^8dQhRRRR/#r9r)rs"rrrs  # # rc\V4pV'gR#\Flwr#VFapVPR4'd<VPR4'd%\P !WA4'dVuu#KUWA9gK]Vuu# Kn R#)zHClassify a failure from error-text keywords; ``unknown`` is the default.unknownz\b)rC_FAILURE_TYPE_ORDER startswithendswithr?search)r:rKkindpatternspatterns& r_classify_failure_typerisq E "D -G!!%((W-=-=e-D-D99W++K,  . rjob_name failure_type output_filec 0V^8dQhRRRRRRRRRRRR /#) rrNr;r:rj Optional[str]rkrlrztuple[str, bool]r)rs"rrrs>)!)! )!)!*7)!N[)!)!)!rc $\T;'gR4p\W4p\V4p\W4p\ 4P 4pT;'g \ V4pVe \V4MRp\4;_uu_4p V PRV34P4p V e.V R,R8Hp V PRWWG34W{3uuRRR4#V PRWpWSWWd34VR3uuRRR4# +'giR#;i) uRecord (or refresh) the incident for ``job_id`` + ``error``; returns ``(incident_id, is_new)``. An existing row for the signature refreshes ``last_seen_at``/``error``/``output_file`` and keeps its state — a ``closed`` incident stays closed, while a ``resolved`` one (job recovered, then broke the same way again) re-opens as ``detected`` so the operator is alerted once more. A changed error text mints a new incident.r>Nz/SELECT id, state FROM cron_incidents WHERE id=?stateresolvedaUPDATE cron_incidents SET last_seen_at=?, error=?, output_file=?, state=CASE WHEN state='resolved' THEN 'detected' ELSE state END, alerted_at=CASE WHEN state='resolved' THEN NULL ELSE alerted_at END, closed_at=CASE WHEN state='resolved' THEN NULL ELSE closed_at END WHERE id=?zINSERT INTO cron_incidents (id, job_id, error_sig, state, failure_type, first_seen_at, last_seen_at, error, output_file) VALUES (?, ?, ?, 'detected', ?, ?, ?, ?, ?)T) r;rXrLr^ _hermes_now isoformatrir7r2fetchone) rNr:rjrkrlsig stored_error incident_idr r,rowreopens &&$$$ rupsert_incidentrzs2 F 6 )C 'Lv+K - ! ! #C@@#9%#@L&1&=#k"4K 4ll = ~ (*  ?\Z/F LL! K= &   >#S  (  D 1 sAC>C>> D c$V^8dQhRRRRRR/#)rrwr;rprboolr)rs"rrrs!""C"""rcV\9dR#\4P4p\4;_uu_4pVP RV34P 4pVeVR,R8Xd RRR4R#VR8XdVP RW 34RRR4R#VR,V8Xd RRR4R#VR8XdVP R W"V34MVP R W34RRR4R# +'giR#;i) uvTransition an incident's lifecycle state; return whether it changed. ``closed`` is terminal for that signature (re-open happens by a changed error minting a NEW incident). Unknown states are rejected (no-op, ``False``). ``alerted`` also stamps ``alerted_at`` — every time, so the cooldown reminder (see ``cron.scheduler._upsert_incident_for_failure``) restarts its window.Fz+SELECT state FROM cron_incidents WHERE id=?NrpclosedalertedzBUPDATE cron_incidents SET state='alerted', alerted_at=? WHERE id=?TzUPDATE cron_incidents SET state='closed', closed_at=?, acked_at=? WHERE id=? AND state != 'closed'z,UPDATE cron_incidents SET state=? WHERE id=?)INCIDENT_STATESrrrsr7r2rt)rwrpr r,rxs&& rset_incident_staters  O# - ! ! #C 4ll 9K> (*  ;#g,(2  I  LLT"   w<5   H  LL7;'   LL>$ 5 s5C/7C/C/50C// D c V^8dQhRRRR/#)rrwr;rr|r)rs"rrrs55c5d5rc\VR4#)zJAcknowledge (close) an incident; ``False`` when missing or already closed.r~)r)rws&r ack_incidentrs k8 44rc V^8dQhRRRR/#)rrNr;rintr)rs"rrrs))c)c)rc  \4P4p\4;_uu_4pTPRT\ T;'gR434p\ VP ;'g^4uuRRR4# +'giR#;i)a\Mark every open incident for ``job_id`` ``resolved`` after a successful run; returns how many. Without this the ledger only ever grows: a one-off failure (a config drift skip, a provider outage) stayed ``detected``/``alerted`` forever after the job recovered, so ``hermes cron incidents`` showed dozens of "open" incidents for jobs that had been green for weeks (32 of 32 on one install). ``resolved`` is distinct from the operator's ``closed`` on purpose: a repeat of the same error re-opens a resolved incident and alerts again (see ``upsert_incident``), whereas ``closed`` keeps that signature silent.zzUPDATE cron_incidents SET state='resolved', closed_at=? WHERE job_id=? AND state IN ('detected', 'alerted')r>N)rrrsr7r2r;rrowcount)rNr r,cursors& r!close_incidents_for_recovered_jobrsf - ! ! #C 4 F #fll# $  6??''a( s?A<+A<< B c V^8dQhRRRR/#)rrprnrztuple[str, tuple]r)rs"rrrsGGG+<GrcVfR#RV33#)Nz WHERE state=?)r>rr)rps&r _state_filterrs}8F+;eX*FFrc V^8dQhRRRR/#)rrprnrzlist[dict[str, Any]]r)rs"rrrs ' '- '3G 'rc6VeV\9d.#\V4wr\4;_uu_4pVPRV,R,V4P 4pRRR4XUu.uFp\ V4NK up# +'giL+;iuupi)zFReturn incidents, newest-activity first, optionally filtered by state.NzSELECT * FROM cron_incidentszB ORDER BY julianday(last_seen_at) DESC, last_seen_at DESC, id DESC)rrr7r2fetchalldict)rpwhereparamsr,rowsrxs& rlist_incidentsrs U/9 !%(ME 4|| *U 2R S   (*  "& &#DI &&  's/B,B B c V^8dQhRRRR/#)rrwr;rzOptional[dict[str, Any]]r)rs"rrrs22c2&>2rc\4;_uu_4pVPRV34P4pRRR4Xe \V4#R# +'giL!;i)z'SELECT * FROM cron_incidents WHERE id=?N)r7r2rtr)rwr,rxs& r get_incidentrsL 4ll 5 ~ (*  491T1 s "A A c V^8dQhRRRR/#)rrprnrrr)rs"rrrs33=3C3rcVeV\9d^#\V4wr\4;_uu_4pVPRV,V4P 4pRRR4Xe\ VR,4#^# +'giL(;i)Nz(SELECT COUNT(*) AS n FROM cron_incidentsn)rrr7r2rtr)rprrr,rxs& rcount_incidentsrsj U/9!%(ME 4llEMvV__a O3s3x=22 s (A99 B )detectedrrqr~)) rate_limit)z\b429\bz rate limitz usage limitquota)timeout)rz timed out)auth)z\b401\b unauthorizedauthenticationr)delivery)rdeliver delivering)config)r configuration validation)script)rno_agent)agent)rmodelprovider inference)N)5__conditional_annotations____doc__ __future__rrSr?sqlite3 threading contextlibrpathlibrtypingrrrrr rr rhermes_constantsr hermes_timer rrr __annotations__rrbrJrRcompilerQRLockr6rr*r(r7rCrLrXr^rirzrrrrrrr)rs@rrs # %66*,*#'&@ zzQ  B n>A "R ' )!:>)!^b)!!%)!X"J5 )$G '233r