+ jHj.TRt^RIHt^RIt^RIt^RIt^RIt^RIt^RIH t ^RI H t H t H t ^RI Ht^RIHtHtHt]P&!]4tRRltR R ltR:R R lltR RltRRltRRltRRltRRltRRltRRltRRlt R:RRllt!Rt"R t#R!t$R"t%R#t&R$R/R%R&llt'R:R'R/R(R)lllt(R*R+lt)R,R-/R.R/llt*R:R,R0/R1R2lllt+R3RR']&/R4R5llt,R;R6R7llt-R8R9lt.R#)/state/gateway.lifecycle.json`` closes the gap: :func:`record_startup` finds ``phase == "running"`` from the previous life → unclean death, appended to the exit-diag log as ``gateway.previous_unclean_exit`` and logged at WARNING; :func:`mark_exited` rewrites ``phase=exited`` on every clean exit path. Best-effort: forensics must never affect the lifecycle. ) annotationsN)closing)datetimetimezoneUTC)Path)AnyDictOptionalcV^8dQhRR/#)returnr)formats"=/home/ubuntu/.hermes/hermes-agent/gateway/lifecycle_ledger.py __annotate__rsiidic^RIHpHp\PP RR4P 4'dV!4#V!4#)zEHERMES_HOME for process-level identity files (ignore task overrides).get_hermes_homeget_process_hermes_home HERMES_HOME)hermes_constantsrrosenvirongetstriprs r_process_hermes_homers7I)+ }b(I(O(O(Q(Q " $hWfWhhrc$V^8dQhRRRRRR/#)r homeOptional[Path]relativestrr rr)rs"rrr#s&RR^RRRrc>Vf \4MTP!V!#N)rjoinpath)r r"s&*r _home_pathr'#s&*l " F F QQrc V^8dQhRRRR/#)r r r!r rr)rs"rrr's??n??rc\VRR4#)z6Return ``/state/gateway.lifecycle.json``.statezgateway.lifecycle.json)r'r s&rget_lifecycle_sentinel_pathr,'s dG%= >>rcV^8dQhRR/#)r r r#r)rs"rrr,s))#)rcR\P!\4P4#r%)rnowr isoformatrrr_now_isor1,s <<  & & ((rc$V^8dQhRRRRRR/#)r pathr#wantedzdict[str, str]r zdict[str, int]r)rs"rrr0s!  s N ~ rc/p\VRR7;_uu_4pVFhpVPR^4^,pWQ9gK$\VP4^,4W!V,&\V4\V48XgKhM RRR4V# +'giT#;i \\ \ 3d/u#i;i)zN``{dst: int}`` for each ``src: dst`` key found in a ``Key: value`` /proc file. utf-8-sigencoding:N)opensplitintlenOSError ValueError IndexError)r3r4foundfhlinekeys&& r _proc_fieldsrE0sE  $ - -jja(+=),TZZ\!_)=E+&5zS[0 . L. - L Z , s:B&"B?BBB& B# B&#B&&CCcV^8dQhRR/#)r r dict[str, Any]r)rs"rrr@s  ~ rc \RRR/4p\RRRRRR R R R /4pVPR R 4VPR R 4r2VPV4VeVe W#, VR &V#) zCheap /proc snapshot (KiB): own RSS + MemTotal/MemAvailable + swap used. Linux-only (``{}`` elsewhere), never raises; the 30s heartbeat embeds it so OOM cycles are classifiable.z/proc/self/statusVmRSSrss_kibz /proc/meminfoMemTotal mem_total_kib MemAvailablemem_available_kib SwapTotalSwapFreeN swap_used_kib)rEpopupdate)samplemem swap_total swap_frees r sample_memoryrX@s-/C DF _nVi)4k:z)[ \CGGK6 D8Q  MM#)"7","8 Mrc V^8dQhRRRR/#)r r3rr Optional[dict[str, Any]]r)rs"rrrMs44T464rc\P!VPRR74p\ T\ 4'dT#R# \\3dR#i;i)r6r7N)jsonloads read_textr>r? isinstancedict)r3datas& r _read_jsonrbMsRzz$..+.>?dD))43t3 Z s&AAAc$V^8dQhRRRRRR/#)r payloadrGr r!r Noner)rs"rrrUs& J J^ J> Jd Jrc^RIHp\V4p^RIHpV!VP 4V!W0RR7R# \ d\PRRR7R#i;i))atomic_json_writemkdir_under_hermes_homeN)indentz"Failed to write lifecycle sentinelTexc_info) utilsrhr,rrjparent Exceptionloggerdebug)rdr rhr3rjs&& r_write_sentinelrsUsP J+*40< ,$5 J 9D IJs37"AAc$V^8dQhRRRRRR/#)r recordrGr r!r rer)rs"rrrbs& L Ln LN Lt Lrc x\VRR4p^RIHpV!VP4VP RRR7;_uu_4pVP \ P!V\R7R,4R R R 4R # +'giR #;i \d\PR R R 7R #i;i) zVAppend a JSON line to gateway-exit-diag.log (same format as the CLI's ``_exit_diag``).logszgateway-exit-diag.logriazutf-8r7)default Nz$Failed to append unclean-exit recordTrl) r'rrjror:writer\dumpsr#r>rqrr)rur r3rjrBs&& r_append_exit_diagr}bsL$(?@< , YYsWY - - HHTZZ4t; <. - - - L ;d KLs0AB3A?5B? B BB"B98B9c(V^8dQhRRRRRRRR/#)r pidr start_time create_timer boolr)rs"rrros(34rct\V4p^RIHpV^8:gV!V4'gR#^RIHpT!T4pTfR#\ T4\\39d\T\T4, 4R8*#\ T4\\39dT\T4R,8*#R# \dR#i;i)uTrue when ``pid`` is a live process that is the sentinel's incarnation — guards the ``--replace`` race: a live matching owner mid-teardown is a handover, not a death. Identity is the psutil ``create_time`` the sentinel stamps at claim (epoch seconds, same producer on both sides). The sentinel's ``start_time`` is the ledger claim time and is NOT comparable with ``gateway.status.get_process_start_time`` (proc ticks on Linux, centiseconds elsewhere) — the old comparison never matched, so every ``--replace`` handover read as an unclean death. A pre-stamp sentinel (no ``create_time``) still has ``start_time`` in epoch seconds: the owner was born BEFORE it claimed, a PID reuser AFTER the owner died, so a birth later than the claim is a reuser.) _pid_existsF_process_create_timeTg@) r<gateway.statusrrphermes_cli.process_identityrtypefloatabs)rrrpid_intrractuals&&& r_pid_is_sentinel_ownerrosc(. a<{733 4A !' *F ~ KS%L(6E+../366 JC<'z*S000  s%B(( B76B7c V^8dQhRRRR/#)r rUrGr rr)rs"rrrs   4 rc^RIHpHpVPR4VPR4rC\ V\ 4'gR#WB8;'g1\ V\ 4;'dV^8;'d WC, V8#)zHeuristic only (classification stays with the reader); thresholds are memory_status' "critical" tier so a live warning and a post-mortem verdict agree.)_CRITICAL_AVAILABLE_FRACTION_CRITICAL_AVAILABLE_KIBrLrNF)gateway.memory_statusrrrr_r<)rUrrtotalavails& r_suspected_oomrsm\77?+SWW5H-I5 eS ! !  *  5#]]519]]A]1]rc V^8dQhRRRR/#)r r r!r rZr)rs"rrrsn8Prc\\V44pV'dVPR4R8wdR#\VPR4VPR4VPR44'dR#RVPR4RVPR 4R VPR4/p^R IHp\V!V44;'g/pV'dVPR 4VR &VPR4p\V\4'dWRR&\V4'dRVR&V# \ d/pLni;i)zNEvidence dict when the previous life died uncleanly, else ``None``. Read-only.phaserunningNrrr prior_pidprior_started_at started_atprior_start_time)get_loop_heartbeat_path updated_atlast_heartbeat_atrUlast_heartbeat_memT suspected_oom) rbr,rrgateway.shutdown_watchdogrrpr_r`r)r sentinelevidencerhbrUs& rdetect_unclean_exitrs5d;XHLL6 H E /5 6 < <" (*|(<$% &&-C#t),%& #  (,H_ % O  s-D- D-- D=<D= state_db_unclean_integrity_checkg @igN@g @ deadline_atc$V^8dQhRRRRRR/#)r connzsqlite3.ConnectionrOptional[float]r rer)rs"rrrs(KK);K_KhlKrcaaa^RIHoS!\\R7\P !4oRVVV3RllpVP V\4R#)aClaim the integrity-check lease and renew it from a SQLite progress handler. ``report_startup_progress`` is a no-op when no watchdog is armed and never raises. Without ``deadline_at`` the handler always returns 0 -- it must never abort the verdict PRAGMA. With ``deadline_at`` (a ``time.monotonic`` timestamp) it returns non-zero once that instant is reached, so SQLite raises ``OperationalError: interrupted`` and the caller can bound the scan. Synchronous by design: no checker worker thread can outlive the check. )report_startup_progressrcV^8dQhRR/#)r r r<r)rs"rr4_install_integrity_check_lease..__annotate__s#rc<\P!4pVS, \8dVoS!\\R7Se VS8d^#^#)r)time monotonic_INTEGRITY_CHECK_LEASE_RENEW_S_INTEGRITY_CHECK_LEASE_S_INTEGRITY_CHECK_LEASE_PHASE)r/r last_renewrs r _on_progress4_install_integrity_check_lease.._on_progresssAnn  = =J #$@4#S#Sdg#Src2\VR4pVP4'gR#RpVe1\P!4\ R\ V44,p\\P!\V444;_uu_4p\WCR7VPR4P4pRRR4X'd T^,fR #\T^,4# \ \3dRpLi;i +'giLP;i \dypTef\!T\P"4'dFR\T4P%49d(Te\ R\ T44MRpRTR R 2uRp?#R T 2uRp?#Rp?ii;i) uI``"ok"``, ``"absent"``, or the first ``quick_check`` complaint. Never raises. Only after an unclean death — SIGKILL mid-WAL-checkpoint can leave half-written b-tree pages. ``quick_check(1)`` stops at the first problem (~2s on a healthy 500MB store): cheap once per unclean boot, too costly every boot. Opened normally: a WAL store needs its -shm sidecar for read-only, and the PRAGMA writes nothing. Holds a phase-owned startup-watchdog lease for the check duration (renewed from a SQLite progress handler, #115542) so a long healthy check on a huge store is not mistaken for a parked deadlock. ``deadline_s`` (optional): abort the scan once that many seconds elapse and return ``"check-incomplete: exceeded s"``. ``deadline_s=None`` (default) is the historic unbounded behaviour -- the progress handler never aborts the PRAGMA. state.dbabsentNg)rzPRAGMA quick_check(1) interruptzcheck-incomplete: exceeded gszcheck-failed: zcheck-failed: no result)r'existsrrmaxr TypeErrorr?rsqlite3connectr#rexecutefetchonerpr_OperationalErrorlower)r rr3rrrowexclimits&$ rcheck_state_db_integrityrsE dJ 'D ;;==#'K ..*SeJ6G-HHK & W__SY/ 0 0D *4 I,,67@@BC1-03q6> $Rs3q6{R:& K 1 0 &  # 38P8P(Q(Q3s8>>#333=3ICU:./sE0q ; ;u%% &sZ/C'0D ,D8D'C=<C= D DD FA#FFF FFc V^8dQhRRRR/#)r r3rr z Optional[str]r)rs"rrrs   - rc(^RIHp\\P!\ V4RR74;_uu_4pV!V4;'gRuuRRR4# +'giR#;i \ d\PRRR7R#i;i)zOn-disk journal mode of ``path`` via one ``PRAGMA journal_mode`` read on a short-lived connection (no lock outlives it); ``None`` when undeterminable. Never raises.)_on_disk_journal_modeg?)timeoutNz$Could not read state.db journal modeTrl) hermes_state_walrrrrr#rprqrr)r3rrs& r_state_db_journal_modersg: W__SY< = =(.66$> = = =  ;d Ks.8A+A A+ A( "A+(A++"BBdefer_integrity_checkFc(V^8dQhRRRRRRRR/#) r rrGr r!rrr rer)rs"rrrs*>cgtxrc V'dRVR&M-\VR7;q0R&VR9d\PRV4\R\ 4RRR\ P !4/VCV4\PR VPR 4VPR 4VPR 4VPR 4VPRR44R#)aPersist the exit-diag record and log at WARNING. When ``defer_integrity_check`` is set, the forensic store scan is NOT run here (it would gate platform startup); the record carries ``state_db_integrity: "deferred"`` and :func:`run_deferred_integrity_check` runs the scan post-connect. deferredstate_db_integrityr+state.db FAILED integrity check after an unclean gateway exit: %s — sessions may read as missing until it is repaired. Run `hermes doctor`.tstagzgateway.previous_unclean_exitru$Previous gateway life (pid=%s, started_at=%s) exited UNCLEANLY (no exit path ran — SIGKILL / OOM / VM death, or a process kill issued by the agent or one of its descendants, e.g. a pkill/taskkill of the host interpreter image; see #113667). last_heartbeat_at=%s last_mem=%s suspected_oom=%srrrrrFNokr) rrqerrorr}r1rgetpidwarningr)rr rverdicts&&$ r_report_unclean_exitrs)3%&4LQU3VV/0 * * LLE  tXZ0OQVXZXaXaXcpgoprvw NN f  [!8<<0B#CX\\ReEf )*HLL%,P rTc$V^8dQhRRRRRR/#)r r r!rrr rZr)rs"rrr7s"$$$$$Zr$rc  Rp\V4pVe\W VR7RRR\ P !4R \P!4R \4/p^R I H pT!\ P !44pTeYSR &Te"RTR &TPR4'dRTR&\Y04T# \d\P RRR7Li;i \d\P RRR7T#i;i)a\Boot entry point: report any unclean previous exit (evidence dict, also persisted to ``gateway-exit-diag.log`` and logged at WARNING) then claim the sentinel. Never raises. ``defer_integrity_check`` defaults to True: the state.db integrity scan is forensic and must NOT gate platform startup (a large store's ``quick_check`` blocked the ``:8644`` fleet peer-DM transport for minutes). In deferred mode the exit-diag record is written immediately with ``state_db_integrity: "deferred"`` and the scan runs post-connect via :func:`run_deferred_integrity_check`. Pass False for the historic INLINE scan. N)rzUnclean-exit detection failedTrlrrrrrrrprior_unclean_exitrprior_suspected_oomz"Failed to claim lifecycle sentinel) rrrprqrrrrrr1rrrrs)r rrclaimrrs&$ rrecord_startupr7s*.HE&t,   G\ ]J!()UBIIKW[W`W`Wbdprzr| } E*299;7  "#.-   *.E& '||O,,/3+,$ O+ E 4t DE& J 9D I OJs)B4BC"C4"CC"DDr c(V^8dQhRRRRRRRR/#) r rrZr r!rrr r#r)rs"rrr^s.8484&841?8484 84rc\VR4pVP4'd \V4MRpVR8wd3RT;'gR R2p\P RY4;'gRV4M \ WR7pVR9d.VP R4'g\PR V4R \4R R R RRV/p\V\4'd(VPR4eVPR4VR&\Wa4\V\4'dWPR&V# \d\PRRR7R#i;i)uRun the deferred post-connect state.db scan and append a second diag record. Companion to :func:`record_startup` with ``defer_integrity_check=True``: the forensic scan is moved off the startup path (where, on a multi-GB store, it gated every platform adapter — including the ``:8644`` fleet peer-DM transport — for minutes) into a daemon thread started only after platforms connected. ``deadline_s`` bounds a pathological store so the thread cannot be pinned forever. Appends ``{"tag": "gateway.state_db_integrity", "deferred": true, "state_db_integrity": , "prior_pid": ...}`` and logs the existing ERROR text on a bad verdict. NEVER raises -- a forensic scan must not disturb a running gateway (the alerting path is the bounded per-poll readiness probe ``gateway/readiness.py::_probe_state_db``). Only a WAL store is scanned: in rollback-journal mode (``database.journal_mode: delete``, or a silent WAL fallback) the scan's SHARED lock would block every live commit for its whole run, so the scan is skipped with a ``skipped: ...`` verdict and one WARNING. The on-disk mode decides, not config; an undeterminable mode is skipped too. rwalz.skipped: rollback-journal store (journal_mode=unknownz), scan would block live writeszPrevious gateway exit was unclean, but the state.db integrity check was skipped: %s is a rollback-journal store (journal_mode=%s) and a background scan would block live writes. To check it, stop the gateway and run: sqlite3 %s 'PRAGMA quick_check')r rzskipped:rrrzgateway.state_db_integrityrTrrz(Deferred state.db integrity check failedrlz"check-failed: deferred scan raisedr)r'rrrqrr startswithrr1r_r`rr}rprr)rr rr3moderrus&$$ rrun_deferred_integrity_checkr^s@,"4$ +/3{{}}%d+% 5=GHYHYPYGZ[66G NNY''i  /DPG * *73E3Ej3Q3Q LLE  (* /  ' "  h % %(,,{*C*O"*,,{";F; &' h % %-4) * 4 ?$ O34s$?D D !2D B D "EEc(V^8dQhRRRRRRRR/#) r exit_codez Optional[int]reasonr#r r!r rer)rs"rrrs0PP=PPZhPtxPrc \\V44pVe,VPR4\P!48wdR#RRR\P!4RVRVR\ 4/pR F(pVfK VPV4fKW5,WE&K* \ WB4R# \d\PRR R 7R#i;i) u<Mark the current life as cleanly exited. Idempotent, never raises. Only rewrites a sentinel provably owned by this process: during ``--replace`` the replacement claims it before the old process finishes teardown and must not be clobbered. ``pid=None`` / malformed sentinels have unknown ownership → left alone. Nrrexitedr exit_reason exited_atz(Failed to mark lifecycle sentinel exitedTrl)rr) rbr,rrrr1rsrprqrr)rrr rrrDs&&& r mark_exitedrs P9$?@  HLL$7299;$F ")8UBIIKV_anpv"-xz";1C# S(9(E&m 1 % P ?$ OPs$AB'-B'7B' B''"C  C c V^8dQhRRRR/#)r profile_homerr r#r)rs"rrrsrc\\V44;'g/PR4pRRRR/PVR4# \dR#i;i)z``clean``/``unclean``/``unknown`` for the profile's last gateway life; annotates ``container-boot.log``. ``running`` is unclean: the old PID namespace is gone at container boot.rrcleanruncleanr)rbr,rrp)rrs& rread_prior_exit_labelrsY7 EFLL"QQRYZ'9i8<rs #   ,,&&   8 $iR? )  4 J L@ L B !(!% &K_cK8#S[_#S#SL jo8$QU$$N84BF84584vP.r