߲jTUdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl m Z ddlmZmZmZej$eZdZdZd Zd Zd Zd Zehd ZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%ejLZ'da(de)d<d)dZ*d*d+dZ+d,dZ,d-dZ-d.dZ.d/dZ/d0dZ0d1dZ1Gd d!Z2 d*ed" d2d#Z3d3d$Z4d4d%Z5d5d6d&Z6d7d8d'Z7d3d(Z8y)9uStartup-liveness watchdog — respawn a gateway that wedges before its loop runs (OOF-298). Every other liveness backstop (loop-liveness watchdog, shutdown watchdog, heartbeat file) assumes startup succeeded; none can fire if the process deadlocks before the event loop is alive (OOF-298: ~30h with every thread in ``futex_wait_queue``, zero logs, s6 saw a live PID). A daemon thread armed at process entry and disarmed once the loop is confirmed live dumps all-thread stacks (``faulthandler``), records the exit in the lifecycle ledger (NS-608) and ``os._exit``\ s with the service-restart code so s6/systemd respawn. Slow-but-alive startups are not killed. Order of authority: (1) phase-owned progress leases (:func:`report_startup_progress`) — authoritative, prove the *startup path itself* is alive, work for I/O-bound phases (schema migration, corruption repair in ``SessionDB.__init__``) with ~zero CPU; (2) process-wide CPU progress, capped at ``_MAX_CPU_EXTENSIONS`` since an unrelated thread burning CPU must not hide a parked startup thread forever. Known limitation: a *spinning* startup deadlock earns the capped extensions before firing; the observed class is parked-thread deadlocks. Idle-by-design waits call :func:`kick_startup_watchdog` (respawn-storm backoff, up to 300s); MCP discovery's 120s wait sits inside the 300s default. IMPORT-LIGHTNESS IS A CORRECTNESS PROPERTY: top-level module, stdlib only. Arming must precede importing ``gateway`` (hundreds of modules; an import-time deadlock is in scope), and at fire time the wedged main thread may hold the import lock — so the fire path does no imports on its own thread; the ledger write runs on a helper thread joined with a timeout. Config is env-only (``HERMES_STARTUP_WATCHDOG=0``, ``HERMES_STARTUP_WATCHDOG_TIMEOUT_S``) because config.yaml parsing is itself in scope. Everything is best-effort: a watchdog failure must never affect the startup it observes. ) annotationsN)datetimetimezone)Path)AnyDictOptionalgr@g>@KHERMES_STARTUP_WATCHDOG!HERMES_STARTUP_WATCHDOG_TIMEOUT_S)logszgateway-startup-watchdog.log>0noofffalseg@g?g @g$@armeddisarmedfiringz!Optional['StartupWatchdogHandle']_handlectjjddj}|r t |St j dk(rZtjjddj}|r t |nt jdz dz }|dz St jdz S) uYHERMES_HOME for diagnostic files — stdlib-only replica of the hermes_constants default. HERMES_HOMEwin32 LOCALAPPDATAAppDataLocalhermesz.hermes)osenvirongetstriprsysplatformhome)val local_appdatabases /logs/gateway-startup-watchdog.log``.)r*joinpath_DUMP_RELATIVE)r%r(s r)get_startup_watchdog_dump_pathr/qs##4)=)?D 4==. ))r+ctjjtdj j t vS)z:True when ``HERMES_STARTUP_WATCHDOG`` opts out explicitly.r)rr r!ENV_STARTUP_WATCHDOGr"lower_FALSEYr+r)startup_watchdog_disabledr5ws/ ::>>. 3 9 9 ; A A Cw NNr+c.tjjtdj }|st S t |}|dkrt St|tS#t$r)tjdt|t t cYSwxYw)zEDeadline in seconds; env override, floor-clamped, default on garbage.rz/Ignoring non-numeric %s=%r; using default %.0fsr) rr r!ENV_STARTUP_WATCHDOG_TIMEOUT_Sr""DEFAULT_STARTUP_WATCHDOG_TIMEOUT_Sfloat ValueErrorloggerwarningmax_MIN_TIMEOUT_S)rawvalues r) resolve_startup_watchdog_timeoutrA|s **..7 < B B DC 112c  z11 un %% 2 = *C1S 21 2s A""/BBc t}|jjddt|dd5}||dddy#1swYyxYw#t$rt j |dYywxYw)zQOpen the dump file for append and hand it to *write*; failures only log at DEBUG.T)parentsexist_okazutf-8)encodingNexc_info)r/parentmkdiropen Exceptionr;debug)write failure_msgpathfhs r) _append_dumprRsn1-/ $6 $g . " "I    1 [4 01s.5A A A AAA A87A8c$tfddy)zDAppend a one-line JSON metadata record beside the faulthandler dump.c^|jtjtdzS)N)default )rNjsondumpsstr)rQrecords r)z$_write_dump_record..s 288DJJvs;dBCr+z,Failed to write startup watchdog dump recordN)rR)rZs`r)_write_dump_recordr\sC6r+cR tt||g|y#t$rYywxYw)zPLog at *level*; never raises (the wedged main thread may hold the logging lock).N)getattrr;rL)levelmsgargss r) _log_quietlyrbs/ s*T*    s  &&cD ddlm}||dy#t$rYywxYw)zRecord the watchdog exit in the NS-608 lifecycle sentinel. Runs on a helper thread (see ``_fire``): the import can block on the interpreter import lock, and the fire path must reach ``os._exit`` regardless. r) mark_exitedstartup_liveness_watchdog)reasonN)gateway.lifecycle_ledgerrdrL) exit_coderds r)_mark_lifecycle_exitris'  8I&AB    s  ceZdZdZddZddZdddZdddZeddZ ddZ ddd Z e dd Z dd Zdd Ze dd Ze ddZddZddZddZy)StartupWatchdogHandlez>+2(037  +/#//+r+c|j5|jtk(r t|_ddd|jj y#1swY$xYw)uStartup reached a live event loop — stand down. Idempotent. Atomic with respect to firing: whichever of disarm/fire takes the state lock first wins, so a disarm landing before the fire sequence is never lost. N)rvrsrr _DISARMEDrysetrs r)disarmzStartupWatchdogHandle.disarmsI    ({{f$'  (   " ( (s AAc tdt|}|j5t j |jz|z|_dddy#ttf$rd}YRwxYw#1swYyxYw)zPush the deadline out to ``now + timeout + extra_s``. For call sites about to block intentionally with ~zero CPU (the respawn-storm backoff sleep), otherwise indistinguishable from a parked deadlock. rmN) r=r9 TypeErrorr:rvrorprnrw)rextra_sextras r)kickzStartupWatchdogHandle.kicksx  U7^,E   G!^^->FDN G G:& E  G GsA*A-A*)A*-A6cb t|}|dkryt|t}|j 5t |jtj|z|_|rt||_ |xjdz c_ dddy#ttf$rYywxYw#1swYyxYw)uClaim a progress lease: this phase expects up to ``expected_s`` more seconds of work. The authoritative "still making progress" signal — owned by the startup path, so it works for I/O-bound phases and cannot be counterfeited by unrelated threads. Clamped to ``_MAX_LEASE_S`` per call so one buggy caller cannot silence the watchdog indefinitely; long phases renew. Never raises. Nr) r9rr:min _MAX_LEASE_Srvr=r|rorprYr}r~)r expected_sphaseexpecteds r)leasezStartupWatchdogHandle.leases Z(H q= x.    # #D$5$5t~~7G(7R SD $'J!    "   # # :&     # #s BAB%B"!B"%B.c(|jtk(SN)rsrrs r)rzStartupWatchdogHandle.disarmeds{{i''r+cV|jduxr|jjSr)rzis_alivers r)rzStartupWatchdogHandle.is_alives#||4'CDLL,A,A,CCr+NcV|j|jj|yy)Ntimeout)rzjoin)rrs r)rzStartupWatchdogHandle.joins& << # LL  g  . $r+cJ tjS#t$rYywxYw)zBProcess-wide CPU time (user+system, all threads); None on failure.N)ro process_timerLr4r+r)_process_cpu_secondsz*StartupWatchdogHandle._process_cpu_secondss( $$& &  s  ""c  tj|jddjt j |jz }tdd|j||j|jttjtj j#dt%j&|jt)|d|j|j*|j,|jd t/d d  t1j2d  tjt8|jfdd}|j|j;t<|j>jA|jC|jy#t$rYwxYw#t$rt4j7d d YwxYw#t$rYwwxYw)uForensics, then exit — with the exit itself independently bounded. Everything producing forensics (logging, dump record, faulthandler, ledger) can block: the wedged main thread may hold the logging handler lock, the disk may be hung. None of that may stop the respawn, so the escort thread starts FIRST. ``os._exit`` is async-signal-safe and lock-free by design. Tz$gateway-startup-watchdog-exit-escorttargetdaemonnamecriticalzGateway startup did not reach a live event loop within %.0fs (elapsed %.0fs, %d extension(s)), holds no progress lease and shows no CPU progress; dumping all thread stacks and exiting with code %d so the service supervisor can restart it (OOF-298).zstartup_watchdog.firedr) tstagpidrn elapsed_s extensions lease_countlast_lease_phaserhc0tj|dS)NT)file all_threads) faulthandlerdump_traceback)rQs r)r[z-StartupWatchdogHandle._fire..6s|22Mr+z4Startup watchdog file-based faulthandler dump failed)rz)Startup watchdog faulthandler dump failedrGzgateway-startup-watchdog-ledger)rrarrrN)"rtThread _exit_escortstartrLrorprqrbrnr{rhr\rnowrutc isoformatrgetpidroundr~r}rRrrr;rMrir_LEDGER_JOIN_TIMEOUT_Srr_exit)relapsed ledger_threads r)_firezStartupWatchdogHandle._fires    ((>2CD6M    !   '=  >  4>>"a   B U LLDtL T U   s60F<!G 8AG2< G G  G/.G/2 G>=G>c|jt|jjry|j |j y)zHard-exit if the forensic fire path wedges (bounded-exit seam). Deliberately free of log handlers, filesystem access, module loads and any lock shared with application code: only a sleep, an Event check and the exit seam. N)_sleep_FIRE_EXIT_BOUND_Sris_setrrhrs r)rz"StartupWatchdogHandle._exit_escortLs4 &' ?? ! ! #  4>>"r+c.tj|y)z4Seam for tests; production is a bare ``time.sleep``.N)rosleep)secondss r)rzStartupWatchdogHandle._sleepWs 7r+c.tj|y)z2Seam for tests; production is a bare ``os._exit``.N)rr)codes r)rzStartupWatchdogHandle._exit\s r+c|j5|jtk7r dddy||_ dddy#1swYyxYw)zDSet a new deadline under the state lock; False when no longer armed.NFT)rvrsrrrw)rdeadlines r)_extend_if_armedz&StartupWatchdogHandle._extend_if_armedasE    {{f$  &DN    s <<Ac |j} |j5|jtk7r dddy|j}dddt j z }|dkDr,|jjt|tryt j }|j5|j}|j}ddd|kDri|jt||tt|jzsyt!dd|jxsd||z |j}6|j}||||z nd}||t"k\r|j$t&krr|}|xj$dz c_|jt j |jzsyt!dd|j||j$t&|j5|jtk7r dddyt(|_ddd|j+y#1swYxYw#1swY~xYw#1swY4xYw)Nrrr<ueGateway startup exceeded %.0fs but phase %r holds a progress lease for another %.0fs — honoring it.unknownruGateway startup exceeded %.0fs but is consuming CPU (%.1fs this window); extending the startup watchdog deadline (CPU-fallback extension %d of %d — phases doing long legitimate work should call report_startup_progress instead).)rrvrsrrrwrorprywaitr _POLL_SLICE_Sr|r}rr=rnrb_CPU_PROGRESS_MIN_Sr{_MAX_CPU_EXTENSIONS_FIRINGr) rlast_cpur remainingr lease_until lease_phasecpu window_deltas r)_runzStartupWatchdogHandle._runis7,,.!! *;;&( * * >> *!4>>#33I1}'',,SM5R,S.."C!! 0"// "//  0S ,,ScC W[WeWeDf>f-ghHNNK$<9kC>O  446++-C-0_AU3>[_L'LI II  II#ctj|jdd} |j||_y#t$rt j ddYywxYw)NTzgateway-startup-watchdogrz(Failed to start gateway startup watchdogrGF)rtrrrrLr;rMrz)rthreads r)_startzStartupWatchdogHandle._startsZ!!4F`a  LLN    LLCdL S s< AA)rnr9rhintreturnNonermrr9rrrrr9rrYrrrboolr)rOptional[float]rr)rr)rr9rr)rrrr)rr9rr)__name__ __module__ __qualname____doc__rrrrpropertyrrr staticmethodrrrrrrrrr4r+r)rkrksF,$ # G#*((D/ <#| #8tr+rk)rhc tryt5t#tjrtcdddS|t |dkDr t |n t }t ||}|js dddy|a|cdddS#1swYyxYw#t$rtjddYywxYw)zArm the process-wide startup watchdog. Idempotent; never raises. Returns the (possibly pre-existing) handle, or ``None`` when disabled via ``HERMES_STARTUP_WATCHDOG=0`` or when the thread could not be started. Nrz&Failed to arm gateway startup watchdogTrG) r5 _handle_lockrrr9rArkrrLr;rM)rnrhresolvedhandles r)arm_startup_watchdogrs $ &  "w'7'7'9   (U9-=-Ai 57  +8Y?F==?  G     = MsJ B!B! B B!AB?B!B B!BB!B!! CCc t5t}da||jdddy#1swYyxYw#t$rtj ddYywxYw)uDisarm the process-wide startup watchdog, if armed. Never raises. ``disarm()`` runs while still holding the singleton lock — it is non-blocking, and this closes the window where a concurrent re-arm could swap in a new handle that the disarm then misses. Nz)Failed to disarm gateway startup watchdogTrG)rrrrLr;rMrs r)disarm_startup_watchdogrs\Q  FG!     Q @4 PQs#8,8588 AAc t5t}dddt|||yy#1swYxYw#t$rtj |dYywxYw)zVCall ``handle.(*args)`` on the armed handle; no-op when unarmed, never raises.NTrG)rrr^rLr;rM)methodrOrars r)_with_armed_handlers`1  F    #GFF #T *    1 [4 01s6*636 AActdd|y)zExtend the armed watchdog's deadline. No-op when not armed; never raises. Call before intentionally blocking with ~zero CPU activity (e.g. the respawn-storm backoff sleep) so the idle wait is not mistaken for a parked deadlock. rz'Failed to kick gateway startup watchdogNr)rs r)kick_startup_watchdogrs vH'Rr+c tdd||y)uDeclare a phase-owned progress lease on the armed startup watchdog. Call from startup phases about to do legitimately long synchronous work — most importantly ``state.db`` schema migrations and corruption repair/backup inside ``SessionDB.__init__`` — with an honest worst case, renewing for multi-step phases. Per-call duration is clamped to ``_MAX_LEASE_S``. No-op when not armed; never raises — safe to call unconditionally from application code. rz!Failed to report startup progressNr)rrs r)report_startup_progressrsw CZQVWr+cpt5t}daddd|jyy#1swYxYw)z0Drop the module singleton (test isolation only).N)rrrrs r)_reset_for_testsrs<  s ,5)rrr)r%zOptional[Path]rrr)rr9)rOrYrr)rZzDict[str, Any]rr)r_rYr`rYrr)rhrrr)rnrrhrrzOptional[StartupWatchdogHandle]r)rrYrOrYrrrrrr)9r __future__rrrWloggingrr#rtrorrpathlibrtypingrrr getLoggerrr;r8r>SERVICE_RESTART_EXIT_CODEr1r7r. frozensetr3rrrrrrrrrrrurr__annotations__r*r/r5rArRr\rbrirkrrrrrrr4r+r)rsN8#   '&&   8 $%*"0!D9 / 0         y~~ -1 *1 #* O &$1   ttp"&/% @Q$1S Xr+