߲jLXdZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z m Z ddlmZddlmZmZmZmZddlmZddlmZmZdd lmZej8eZd Zd Z d Z!d Z"d Z#dZ$dZ%dZ&d,d-dZ'GddZ(GddZ)e!f d.dZ*e"e#e$ed d/dZ+d0dZ,d1dZ-d2dZ.d3d2dZ/d4d5dZ0d3d2dZ1ddddd d6d Z2ed! d7d"Z3 d8d#Z4ddd$dd%d& d9d'Z5d:d(Z6d;d)Z7e dddd* d/state/gateway.heartbeat`` so supervisors can tell "process alive" from "loop frozen"; (3) a lifetime thread watchdog that hard-exits when the loop is too frozen to run its own callbacks; (4) a self-rescheduling floor timer that keeps the selector timeout finite.) annotationsN)datetimetimezone)Path)AnyCallableDictOptional)!GATEWAY_SERVICE_RESTART_EXIT_CODE)get_hermes_homeget_process_hermes_home)atomic_json_writegN@g>@g@g$@)statezgateway.heartbeat)logszgateway-shutdown-watchdog.logc\ tt||S#ttf$r|cYSwxYw)z@``max(float(value), floor)``, or ``default`` when not coercible.)maxfloat TypeError ValueError)valuedefaultfloors >/home/ubuntu/.hermes/hermes-agent/gateway/shutdown_watchdog.py _coerce_floatr0s15<'' z "s ++c(eZdZdZddZddZddZy)_LoopFloorTimerHandlezBCancelable owner for the currently scheduled selector floor timer.c^||dc|_|_|_d|_|j y)NF)_loop _interval _cancelled_timer_tick)selfloopintervals r__init__z_LoopFloorTimerHandle.__init__:s(6:He3 DNDO59  c|js6|jj|j|j|_yyN)r!r call_laterr r#r"r$s rr#z_LoopFloorTimerHandle._tick?s.**// KDKr(c`d|_|j|jjyy)NT)r!r"cancelr,s rr.z_LoopFloorTimerHandle.cancelCs( ;; " KK    #r(N)r%asyncio.AbstractEventLoopr&rreturnNone)__name__ __module__ __qualname____doc__r'r#r.r(rrr8sL L!r(rceZdZddZy)_LoopLivenessWatchdogHandlecz||_|j|j|jc|_|_|_yr*) _stop_eventsetjoinis_alivestop)r$ stop_eventthreads rr'z$_LoopLivenessWatchdogHandle.__init__Js,%.8nnfkk6??+ 49dmr(N)r@threading.EventrAzthreading.Thread)r3r4r5r'r7r(rr9r9Is[r(r9cPt|d}t||dkDr|StS)zAKeep at least one timer pending so selector waits remain bounded.r)rr#DEFAULT_LOOP_FLOOR_TIMER_INTERVAL_S)r%r&ivs r_arm_loop_floor_timerrGOs, x %B R!Vr ]]9\ ]]r()probe_interval probe_timeout max_strikes exit_codectjdfd }tj|dd} |jt|S#t$rt j ddYywxYw) zStart an out-of-loop watchdog that hard-exits after missed probes. The caller (``GatewayRunner._start_loop_liveness_guards``) enforces the ``gateway.loop_watchdog: false`` opt-out.cd} jsytj} j|jtjz} jsJ|tjz }|dks|jt|drn jsJy|jrd}ȉ jry|dz }|kr jrytjt 5tjd|ddd t!j"d  jryt%d t'j( jsxyy#t $rYyt $rtjddYywxYw#1swYxYw#t $rtjd dYwxYw) Nrtimeoutz.Failed to schedule gateway loop liveness probeTexc_infog?zGateway event loop missed %d consecutive liveness probes; dumping all thread stacks and exiting with code %d so the service supervisor can restart it. all_threadsz&Loop liveness faulthandler dump failedloop_liveness_watchdog)wait threadingEventcall_soon_threadsafer< RuntimeError Exceptionloggerdebugtime monotonicis_setmin contextlibsuppresscritical faulthandlerdump_traceback_mark_exited_quietlyos_exit) strikes probe_eventdeadline remainingrKr%rJrHrIr@s r _watchdogz/start_loop_liveness_watchdog.._watchdogbs//./9#//+K ))+//: ~~'-7H '')$t~~'77 >[%5%5c)T>R%5%S!'') !!#  " qLG$  "$$Y/ (`Y( (  V++=  " ,D E HHY M//./99    MX\ ] ( ( ( V EPT U Vs5FG3G  F>F>=F>G  G0/G0Tzgateway-loop-liveness-watchdogtargetdaemonnamez.Failed to start gateway loop liveness watchdogrPNr0)rWrXThreadstartr[r\r]r9)r%rHrIrJrKrnrAr@s````` @rstart_loop_liveness_watchdogruWsv"J( ( R  YtBb cF  'z6 ::  EPT UsA A>=A>ctjt5ddlm}|||dddtjt5ddlm}|tk(rddini}|d d|d d |dddy#1swYMxYw#1swYyxYw) uBest-effort terminal stamp on BOTH lifecycle records before ``os._exit`` skips teardown: the lifecycle ledger (so the next boot names the watchdog, not SIGKILL/OOM) and ``gateway_state.json`` (so ``hermes gateway status`` and every other reader of that file stop seeing ``running`` for a process the watchdog killed — #113372). The runtime-status write goes LAST: it is the record housekeeping refreshes, so nothing may overwrite it after we stamp it.r) mark_exited)reasonN)write_runtime_statusrestart_requestedTdegradedg?) gateway_state exit_reason wait_timeoutr7)rbrcr[gateway.lifecycle_ledgerrwgateway.statusryr )rKrxrwryrestarts rrgrgs   Y '.8If-.   Y 'X72;>_1_&-eg X$&t XOV X XX..XXsA7 "B7BB ctjjddjr t St S)zHHERMES_HOME for process-level identity files (ignore profile overrides). HERMES_HOME)rhenvirongetstripr r r7r(r_process_hermes_homers/(* }b(I(O(O(Q " $hWfWhhr(c||StSr*)rhomes r_homers#4?)=)??r(c:t|jtSr*)rjoinpath_HEARTBEAT_RELATIVErs rget_loop_heartbeat_pathrs 5;  !4 55r(cnt||ntj}t|dz d|dz S)u``/state/gateway.loop-tick..sock`` — PID-suffixed so a stale node from a dead process is never mistaken for this gateway's witness. Served by the loop itself (``_tick_socket_handler``), so an answer proves the loop dispatches; the heartbeat cannot. Served by the gateway loop itself (see ``_tick_socket_handler``): an answer is direct proof that the loop is dispatching, which is exactly the property the heartbeat file lost when its write moved off-loop (#90502). rzgateway.loop-tick.z.sock)intrhgetpidr)rpids rget_loop_tick_socket_pathrs8 S_c"))+ 6C ; %7uE#B BBr(c:t|jtSr*)rr_WATCHDOG_DUMP_RELATIVErs rget_shutdown_watchdog_dump_pathrs 5;  !8 99r()r start_timerextract|}t||ntjt j t jjtjd}|t||d<tjt5ddlm}|x}r||d<ddd|r|j#| t%||d|S#1swY-xYw#t$rt&j)dd Y|SwxYw) zAtomically rewrite the loop-liveness heartbeat file; never raises. ``start_time`` (process start, epoch seconds) lets supervisors detect PID reuse.N)r updated_atr_rr) sample_memorymem)indentz&Failed to write gateway loop heartbeatTrP)rrrhrrnowrutc isoformatr^r_rrbrcr[rrupdaterr\r])rrrrpathpayloadrrs rwrite_loop_heartbeatrs #4 (D&)#biik&R-5\\(,,-G-Q-Q-S,0NN,<>G %j 1    Y '!:/ !3 ! GEN! uN$5 K!! N = M KNsC C(C%( D  D )grace_sc@t|t}t|d|zS)z=Return the wall-clock leash for the shutdown watchdog thread.rD)r!DEFAULT_SHUTDOWN_WATCHDOG_GRACE_S) drain_timeoutrgraces rresolve_shutdown_watchdog_delayrs# '#D EE  ,u 44r(c |jjdddtj|t j tjj|xsid}tjt5t|dd5}|jtj |t" d z|j% t'j(|d |jd |j%ddddddtjt5t*j,jd|ddtjdt*j,j%t'j(ddddy#t$rYywxYw#t$r|jd YwxYw#1swYxYw#1swYxYw#1swYyxYw)z:Best-effort faulthandler + metadata dump before hard-exit.Tparentsexist_okNshutdown_watchdog_fired)eventrdelay_sfired_atsnapshotazutf-8)encoding)rz) --- faulthandler dump (all threads) --- )filerTz%(faulthandler.dump_traceback failed) z--- end dump --- z&Gateway shutdown watchdog fired after z.0fzs (pid=z); dumping all thread stacks. rS)parentmkdirOSErrorrhrrrrrrrbrcr[openwritejsondumpsstrflushrerfsysstderr) dump_pathrrheaderfhs r_write_watchdog_dumprstd;1QX",,x||4>>@hnZ\^F   Y 'iw)OSU FC03``a   ?  ' 'RT B %&     Y '6 A'#O!!# -LN O ##5 66  ? HH= > ? 66sfF+G&>GF:*!G G&4A.G2+ F76F7:GGGGG# G&&G/2G;rRzgateway-shutdown-watchdog) done_event snapshot_fnrKrrrc||ntjt|tdkrSdfd } tj|d|j S#t $rtjddYSwxYw)zArm a daemon-thread hard-exit backstop for a wedged shutdown path: exits quietly if ``done_event`` is set within ``delay_s``, else dumps diagnostics and ``os._exit(exit_code)``. Never raises; returns ``done_event`` for disarming.rcFtjz}tj|krJ jt|tjz drytj|krJ j ry  nd}  n t}t||tjt 5tjd|dddtjtjfD]4}tjt 5|j!ddd6tjt 5ddlm}m}||dddtjt 5ddlm}|ddddt- d t/j0 y#t $r}dt |i}Yd}~Sd}~wwxYw#1swYxYw#1swYxYw#1swYxYw#1swYtxYw) N?rNsnapshot_error)rruhShutdown watchdog fired after %.0fs — forcing process exit (asyncio drain path appears wedged; see %s)r)remove_pid_filerelease_gateway_runtime_lock)drain_log_queueshutdown_watchdog)r^r_rVrar`r[reprrrrbrcr\rdrstdoutrrrrrhermes_loggingrrgrhri) rlrexcrpstreamrrrdelaydonerrKrs rrnz(arm_shutdown_watchdog.._watchdogs>>#e+nn)yyX0@%@#!FyGnn) ;;=  5(3(?{}TH(39X9ZVUXF   + Z OOJKPRX Z Zzz3::. F$$Y/        + + U   ( *  +  + ) 6 C ( ) Y(;< / 5($s)4H 5 Z Z   + + ) )sH G G2(G?H H G/ G**G/2G<?H HH TrozFailed to arm shutdown watchdogrPr0) rWrXrrrsrtr[r\r]) rrrrKrrrrnrrs ``` @@rarm_shutdown_watchdogrs$/:Y__5FD '#D EE z   BG $TBHHJ K G 6 F KGs&A(( B  B cK |jd|jd{tjt5|j dddy77#t$rY@wxYw#1swYyxYw#tjt5|j dddw#1swYwxYwxYww)zAnswer a liveness ping with one byte; never raises. Runs on the gateway loop, so a reply witnesses loop schedulability; the write is a socket-buffer copy (no fsync), immune to the stalls that age the heartbeat.1N)rdrainr[rbrcclose)readerwriters r_tick_socket_handlerr0s Tlln  +  LLN         Z  +  LLN   ssC$A"A A"CA1 C A"" A.+A=-A..A=1A:6C=B=B1( B=1B: 6B==Ccv fdjjdDD]>} tjt |j j ddd@y #tttf$r|jdYjwxYw#t$rtjdd Yy wxYw) a;Unlink loop-tick socket nodes left by dead PIDs (POSIX only; never raises). create_unix_server removes a leftover node at OUR path (os._exit / SIGKILL skip the finally-unlink) but not SIBLING nodes from other dead PIDs. os.kill(pid, 0) is a liveness probe only on POSIX (Windows would TerminateProcess).c3.K|] }|k7s |ywr*r7).0pown_paths r z,_sweep_stale_tick_sockets..DscAUVZbUbacs zgateway.loop-tick.*.sock.rT missing_okz#stale loop-tick socket sweep failedrPN)rglobrhkillrrrsplitr IndexErrorrunlinkr[r\r])rstales` r_sweep_stale_tick_socketsr>s Kc!5!56P!Qc .E .EJJ,,S1"56: . G4 .  - . K :T JKs4'B;A)&B)&BBBB B87B8) interval_srrshould_continuecKt|td}dx}x}} tjdk(r`t |}|j j ddt|tjtt|d{}ntjtdd d{}|jxsgD]k}tjt 5t#|j%x} t&r't)| d k\rt+| d } dddn dddm|du|d} tj0t2||| d{||s ||j7tjt 5|j9d{ddd|5tjt 5|j;ddddyyytj<|d{||s ||j7tjt 5|j9d{ddd|5tjt 5|j;ddddyyyc77#1swYxYw#t $rdx}}t,j/d dYwxYw7#t $rt,j5ddYwxYw7\#1swY\xYw#1swYyxYw77#1swYxYw#1swYyxYw#||j7tjt 5|j9d{7dddn #1swYnxYw|Atjt 5|j;ddddw#1swYwxYwwwxYww)a6Rewrite the loop heartbeat file on a cadence until cancelled / gated off. Runs on the gateway loop so a frozen loop lets the file age for monitors. The fsync write goes to a thread (inline, a stalled filesystem blocked the loop inside its own heartbeat and the liveness watchdog killed it: WSL2 fsync stalls p99 31s / max 112s vs a ~90-120s budget) but is *awaited*, never fire-and-forget: an unawaited task would keep the file fresh while the loop was wedged, and one in-flight write at a time keeps a long stall from queuing a thread per interval. Off-loop, file freshness no longer proves loop schedulability, so this task also arms a loop-scheduling witness (``_tick_socket_handler``, flagged as ``loop_tick_socket``); probes must require the witness to agree before classifying WEDGED.r)rNposixTr)rz 127.0.0.1r)hostportrRuLoop tick socket unavailable — liveness probes will have no loop-scheduling witness and will not escalate on a stale heartbeatrP)loop_tick_socketloop_tick_tcp_port)rrrz$Loop heartbeat write failed off-loopr)rDEFAULT_HEARTBEAT_INTERVAL_Srhrrrrrrasynciostart_unix_serverrr start_serversocketsrbrcr[ isinstance getsocknametuplelenrr\warning to_threadrr]r wait_closedrsleep) rrrrr& tick_servertick_socket_path tick_tcp_port_s_snamers rloop_heartbeat_foreverr MsZ)ESQH 6:9K9"]& 77g 8>   # # ) )$ ) F %&6 7 ' 9 9:N?BCS?T!VVK!( 4 45IP[bc ddK!))/R ((3!BNN,<"<&eDVXYIY(+F1I  "-D!8P] ^E= T''(<Z^.3555*?3D  "    $$Y/ 0!--/// 0+((3=$++t+<==, # --) ) )*?3D  "    $$Y/ 0!--/// 0+((3=$++t+<==, #!Ve &&** m\ $  &&5 T Cd S T0 0 0== *0 0 0==  "    $$Y/ 0!--/// 0 0 0+((3=$++t+<===, #sO!A-J0 J $J0.J /0J0>J#J01O!9L>; KKK L>*+O!L)L*L.#O!L$ O!/L>L! L>+O!L&L$L&#O!=L2 O!L>J0 J0#J- (J00$KO!KO!K L>L>LL>LL O!LO!!L>$L&&L/+O!2L;7O!>,O*N >N ?N  O N "O4O OO  OO!)rD)rrrrrrr1r)r%r/r&rr1r) r%r/rHrrIrrJrrKrr1z%Optional[_LoopLivenessWatchdogHandle])rKrrxrr1r2)r1r)rOptional[Path]r1rr*)NN)rrr Optional[int]r1r) rrrOptional[float]rrrOptional[Dict[str, Any]]r1r)rrrrr1r)rrrrrrr1r2)rrrzOptional[threading.Event]rz&Optional[Callable[[], Dict[str, Any]]]rKrrrrrrr1rB)rzasyncio.StreamReaderrzasyncio.StreamWriterr1r2)rrr1r2) rrrrrrrzOptional[Callable[[], bool]]r1r2)9r6 __future__rrrbrerloggingrhrrWr^rrpathlibrtypingrrr r gateway.restartr hermes_constantsr r utilsr getLoggerr3r\rrrE DEFAULT_LOOP_WATCHDOG_INTERVAL_SDEFAULT_LOOP_WATCHDOG_TIMEOUT_S!DEFAULT_LOOP_WATCHDOG_MAX_STRIKESrrrrr9rGrurgrrrrrrrrrrrr r7r(rrsF^#   '00=E#   8 $%)!#&)##' "& %&!4C!!"[[8[^ #^/4^^Aa:86 :; #:;8=:;:;:; :; + :;zX$i @6 C: !4 .= (@LP2/P55&+5TY56#;6@D64@D:>QR $2M1 1#<171KN11-01Sb1h  K 8W[QU9=9=ET9= 9=2N9=Z^9=r(