+ lHj4fha0t$Rt^RIHt^RIt^RIt^RIt^RIHt^RI H t RRlt RRlt R R lt RFR R lltRFR RlltRtRFRRllt/tR]R&RsRRltRRltRRltRRltRFRRlltRR ltR!R"ltR#R$ltR%R&ltR'R(ltR)R*ltR+R,lt RGR-R.llt!R/R0lt"RHt#Rs$]PJ!4t&R1R2lt'R3R4lt(])!R5]!R6R744t*Rs+]PJ!4t,]PZ!^4t.R8R9lt/R:R;lt0R<R=lt1R>R?lt2R@RAlt3RBRClt4RDRElt5R#)IuSession flush / reaping / orphan sweep / cross-backend heartbeat: exit-flush signal handlers, idle + LRU eviction, orphaned session-row sweep, backend heartbeat refresher. Bodies are rebound onto server.py's globals at install time (method_ctx.bind_module), so they reference server.py globals bare — including the knobs _SESSION_TTL_S, _REAPER_SCAN_S, _EXIT_FLUSH_BUDGET_S and _INCREMENTAL_FLUSH_INTERVAL_S. ) annotationsN) env_float) bind_modulec V^8dQhRRRR/#sessionz dict | Nonereturnbool)formats"?/home/ubuntu/.hermes/hermes-agent/tui_gateway/session_reaper.py __annotate__rsbb bbc\V\4'dVPR4MRp\\ VR4;'d\ VRR44#)zWould :func:`_flush_session_messages` write anything for this session? (The exit flush counts these to tell a complete flush from a budget overrun.)agentN_persist_session_session_messages) isinstancedictgetr hasattrgetattr)rrs& r _session_has_pending_flushrsF%/w$=$=GKK 4E 12``wuFY[_7` aarc V^8dQhRRRR/#rr )r s"r rrs[TrcV'dVPR4MRp\VR4'd\VRR4MRpV'gR#\T;'g/RR7;_uu_4VP V4RRR4R# +'giR#;i \ d\ PRRR 7R#i;i) zBest-effort durable flush of one session's transcript via ``agent._persist_session`` (same marker-deduped contract as ``_finalize_session``: repeated calls never duplicate rows). See #13121. rNrrF)hydrate_secretsTz incremental session flush failedexc_info)rrr_session_profile_runtime_scoper Exceptionloggerdebug)rrsnapshots& r _flush_session_messagesr$s %,GKK E._flush_sessions_before_exit..]sIx!+Ea+HAAxs$ $cV^8dQhRR/#rr Noner )r s"r r1_flush_sessions_before_exit..__annotate___sBB$Brc<\P4S,pSF=p\P4V8dR#SR;;,\V4, uu&K? R#)r9N)r6r7r$)deadlinerbudgetresultsessionss r _run)_flush_sessions_before_exit.._run_sF>>#f,G~~8+ 9 !8!A A  rTzhermes-exit-flush)targetdaemonnamezExit flush persisted %d of %d in-memory session transcript(s) within %.1fs; the rest may have been lost (HERMES_TUI_EXIT_FLUSH_BUDGET_S)) _EXIT_FLUSH_BUDGET_Smaxr,sum threadingThreadstartjoinr!warning)r= flushablerNworkerrKrLrMs& @@@r _flush_sessions_before_exitr]Ts&.%5 !3sH;MF {^F')HIxIIIBB  T$=P QF LLN KKi9$ B 9 y& 2 ) rg?c V^8dQhRRRR/#)rr=r/r rGr )r s"r rrxsldrc a \;_uu_4\P4UUu.uF!wrVPR4'gKW3NK# pppRRR4.o XF~wr\P !\ 4;_uu_4\W4RRR4VPR4;pfKRV\P!4JgKmS PV4K Vf\M \RV4p\P4V,pRV 3RllpV!Wv^, , 4^RIHp V !RR 7V!V4R#uuppi +'giEL;i +'giL;i) aInterrupt every in-flight turn and give it ``budget_s`` to settle, so a running tool call ends with a result the teardown's final persist records. A foreground command runs in its own process group and would outlive the gateway, reparented to init. One still alive halfway through the budget ignored the interrupt's SIGTERM: SIGKILL it then, early enough for its result to land as well (the interrupt's own TERM, 1s, KILL outlasts the SIGTERM path's ~1s grace).r2N _run_threadr4c V^8dQhRRRR/#)runtilr8r rGr )r s"r r-_stop_turns_before_exit..__annotate__s77U7t7rc <SF7pVP\RV\P4, 44K9 R#)r4N)rYrTr6r7)rbtthreadss& r _join&_stop_turns_before_exit.._joins,A FF3sEDNN$445 6r)kill_live_foreground_processesT)r.)r)r*itemsr contextlibsuppressr _interrupt_session_turnrVcurrent_threadappend_EXIT_TURN_SETTLE_SrTr6r7tools.environments.baseri) r=sidrBr2rrerKrJrgrirfs & @r _stop_turns_before_exitrsxs  *3//*;P*;quuY?O8C8*;P G    + + #C 1,]+ +A 8QiF^F^F`=` NN1   %-$4 #c8:LF~~&(H77 (aZ  F"t, (O#Q , +s.EE E E E"E E " E2 zdict[int, Any]_exit_flush_prev_handlersFcV^8dQhRR/#rFr )r s"r rrs:::rc^RIp\PV4pW2PJdR#\P !\ 4;_uu_4\4RRR4\P !\ 4;_uu_4\4RRR4\V4'd V!W4R#VPWP4\P\P4V4R# +'giL;i +'giL;i \ d\^\T4,4Rhi;i)uFlush in-memory sessions, then hand off to the prior handler (uvicorn's graceful shutdown, a supervisor's handler, or the default disposition) — this only *prepends* a bounded flush.N)signalrtrSIG_IGNrkrlr r]rscallableSIG_DFLoskillgetpid SystemExitr0)signumframe_signalprevs&& r _handle_exit_flush_signalrs $ ( ( 0D     Y ' '#% (   Y ' '! (~~ V : NN6?? 3 GGBIIK ( ( ' ( ' :S3v;./T 9 :s% C6 D 1AD6 D D (EcV^8dQhRR/#rr r r )r s"r rrsDrc\'dR#\P!4\P!4JdR#^RIpRpVP VP 3Fjp\P!\\\4;_uu_4VPV4pVP V\4V\V&RpRRR4Kl VsV# +'giK;i)aRInstall chaining SIGTERM/SIGINT flush handlers (main thread only). Called before uvicorn takes over signals: its ``capture_signals()`` saves these as the "original" handlers and re-raises into them after graceful shutdown, so the flush also covers terminations outside uvicorn's serve window. Idempotent; False off-main-thread/on failure.TFN)_exit_flush_handlers_installedrVrn main_threadrwSIGTERMSIGINTrkrl ValueErrorOSError RuntimeError getsignalrrt)r installedrrs r "install_exit_flush_signal_handlersrs &%!)>)>)@@I??GNN3  Wl C C$$V,D NN6#< =04 %f -I D C4 &/"  D C Cs 3C  C cV^8dQhRR/#rr )r s"r rrs77T7rcV\JdR#\V\4'dQ\;QJd*RVP 44F 'dK R# R#!RVP 444#\ VRR4RJ#)Tc38"TFp\V4xK R#5ir()_transport_is_dead)rApeers& r rC%_transport_is_dead..sO8N%d++8NsF_closedN)_detached_ws_transportrFanoutTransportall transportsr) transports&r rrsn**)_--sO 8L8L8NOssOsOsO 8L8L8NOOO 9i .$ 66rc$V^8dQhRRRRRR/#rrrstrrrr r r )r s"r rrs! 8 83 8 8$ 8rc4VPR4'g#\V4'g\W4'dR#VPR4pVe0VP4'gVPR4'gR#\ VPR44#)aVShared hard exemptions for both reapers (the LRU cap applies them WITHOUT the age gate: eligible the moment it loses its client): never evict a session mid-turn, awaiting input, still building, owning live delegated work, or on a live transport. Lazy watch sessions never start a build, so their unset agent_ready must not make them immortal.r2F agent_readylazyr)r_session_pending_kind_session_has_active_delegationsis_setr)rrrreadys&& r _session_is_lru_evictablersp {{9!6s!;!;?^_b?l?l KK &E  F8K8K gkk+6 77rc V^8dQhRRRR/#)rexcludez str | Noner r r )r s"r rrs G G Gt Grc*\;_uu_4\P4UUu.uFwrW8wgK W3NK pppRRR4\;QJdRX4F 'dK R# R#!RX44#uuppi +'giLJ;i)u4No session but ``exclude`` is mid-turn, building, awaiting input, holding live delegations, or on a live transport. A non-forced memory trim holds the GIL (gc.collect) and every glibc arena lock (malloc_trim) for its whole duration — 20-50 s on multi-GB heaps — which stalls the event loop, drops WS clients past the write deadline and interrupts their turns (#58576); this is the moment it costs no other session. The predicate is advisory (a turn can start right after), so the per-session checks — one may read state.db — run outside ``_sessions_lock``.Nc3<"TFwr\W4xK R#5ir()r)rArrrBs& r rC&_sessions_quiescent..sFvVS(00vsFT)r)r*rjr)rrrrBotherss& r _sessions_quiescentrsl )2):M):vscn(3():M  3FvF33F3F3FvF FFN s B A< A< B<B B c$V^8dQhRRRRRR/#rr )r s"r rrs&IIcIDITIrcx\W4;'d)\V4'*;'d\V4'*#)uReaper eligibility: the LRU exemptions, and not the sole receiver of a pending wake-up — an active /loop or /heartbeat, or a running background process that will notify it (closing the session stops the notification poller that delivers both, and ``agent_close`` kills the process). Kept out of ``_sessions_quiescent``: an idle session waiting on a wake-up does not block a memory trim.)r"_session_owns_live_wakeup_schedule*_session_owns_pending_process_notification)rrrs&&r _session_is_reapablersA &c 3 H H<^_fwGGIrc(V^8dQhRRRRRRRR/#) rrrrrrr.r8r r r )r s"r rrs(..s.T..$.rc\VPR4;'gR4p\VPR4;'gR4pW#, \8:gW$, \8:dR#\W4#)zTTL eviction: idle-for-TTL AND older-than-TTL, plus the reap exemptions (the cheap clock check first: the exemptions read the session store, and this runs for every session on each reaper pass). last_activer4 created_atF)r8r_SESSION_TTL_Sr)rrrr.rrs&&& r _session_is_evictablers] M299c:Kw{{<077C8J n,1An0T  --rcV^8dQhRR/#rFr )r s"r rrsXXTXrc \P4p\4\ ;_uu_4\ P4UUu.uFwr\WV4'gKVNK pppRRR4XFp\VRV3RlR7K \4\4\4\4'g\P R4R#^RIHpV!R R 7R# \d\P RRR7Li;iuuppi +'giL;i \d6p\P R \!T4P"T4Rp?R#Rp?ii;i) z)periodic incremental session flush failedTrN idle_timeoutc>\W\P44#r()rr6rvss&&r %_reap_idle_sessions..s.CBQUQZQZQ\.]r end_reason predicatezAidle reaper periodic trim deferred: a session is busy or attached) trim_memoryzidle reaper periodic trim)reasonz&idle reaper memory trim failed: %s: %s)r6r;r r!r"r)r*rjr_close_session_by_id_repair_missing_ws_orphan_reaps_enforce_session_cap_reclaim_orphaned_leasesrhermes_cli.mem_trimrtype__name__)r.rrrBvictimsrexcs r _reap_idle_sessionsrs ))+CQ %.__%6]%663:OPSX[:\33%6]  N),] _$%   XYX367' Q @4 PQ^ $ X =tCy?Q?QSVWWXsL C$DD !D 'DD%$"D D  D D" % E%0*E  E%cV^8dQhRR/#rFr )r s"r rr!s***rc \^8:dR#\;_uu_4\P4Uu.uFQpVP R4\ JgK\ VP R44'gK@VP R4NKS ppRRR4XFp\V4K \;_uu_4\P4UUu.uF'wr0\V4'gKV\9gK%VNK) pppVFp\V4K RRR4R#uupi +'giL;iuuppi +'giR#;i)a?Re-arm detached sessions whose disconnect path lost its teardown timer. A resident record otherwise vouches for its lease during every orphan sweep, while the live process prevents PID pruning. Reusing the normal WS grace path preserves reconnect and in-flight-work protections instead of stealing the lease directly. Nr) _WS_ORPHAN_REAP_GRACE_Sr)r*r+rrr_close_sessions_for_transportrj_ws_session_is_detached_pending_ws_reaps_schedule_ws_orphan_reap)rclosed_transportsrrrmissings r rr!s!# ENEUEUEWOEW' ' K 8@V V6!3GKK 4L!M6W[[5EWO ' %i0' $-OO$5 $5LC&w/ 47?P4P C$5  C $S )  O   sRD!DD5D D!6D:D4 (D4 5D4 ;D:D!! D1 4D:: E cV^8dQhRR/#rFr )r s"r rr=s E E$ Erc,\4^RIHpV!\ 44;p'd\PRV4R#R# \d\PRRR7L]i;i \d\PRRR7R#i;i)zEHand the registry the lease ids we still own so it can drop the rest.z!stale deferred lease sweep failedTr)release_orphaned_leasesz-Reclaimed %d orphaned active-session lease(s)zorphaned lease reclaim failedN)_reap_stale_deferred_leasesr r!r"hermes_cli.active_sessionsr_own_live_lease_idsinfo)rdroppeds r rr=sI $%EF-.A.CD D7 D KKG Q E I 84 HI E 4t DEs" A4A-"A*)A*-"BBcV^8dQhRR/#)rr r0r )r s"r rrPs  C rc4^RIHp\4;'g/pVPR4pVf9\ VPR4;p\ 4'dVPR4pV!VRR7pV'd \ V4#^# \d^#i;i))coerce_max_concurrent_sessionsmax_live_sessionsgatewaykey)rr _load_cfgrrrr0r )rcfgraw gateway_cfgcoerceds r _max_live_sessionsrPs MkRgg)* ;:SWWY5G&GkNN//"56C0:MN&s7|-A- s#BA!B; BB BBcV^8dQhRR/#rFr )r s"r rr]snndnrc 6\4pV^8:dR#\;_uu_4\\4V8:d RRR4R#\P 4UUu.uFwr\ W4'gKW3NK pppRRR4XP RR7VFOwr\;_uu_4\\4V8:d RRR4R#RRR4\VRV3RlR7KQ R#uuppi +'giL;i +'giLA;i)rNcX\V^,PR4;'gR4#)r@rr4)r8r)kvs&r r&_enforce_session_cap..fs%1 -(@(G(GC"Hrr lru_evictc\W4#r()rrs&&r rrls K_`bKlrr)rr)lenr*rjrsortr)caprrrB evictable_ss r rr]s  C ax  y>S  -6OO,=^,=&#AUVYA]XcX,= ^  NNHNI ^9~$^$  Kc3l n _ ^s5C5C5C/ 3C/ :C5-D/C55 D  D c(V^8dQhRRRRRRRR/#)rdelayr8fail_logrlevelr rGr )r s"r rros)   S  SW rc|aaaRVVV3Rllp\P!W4pRVnVP4R#)zPRun ``fn`` once on a daemon Timer after ``delay``; log (never raise) on failure.cV^8dQhRR/#rFr )r s"r r*_reaper_daemon_timer..__annotate__qs<<$._runqs0 < D < FE "8d ; ; .sJZ [[rcV^8dQhRR/#)rr r8r )r s"r rrs""U"rc\\RR4f\P4\n\P#)zWall-clock time this process started (first-call time is a good-enough proxy: the heartbeat refresher runs after the gateway is fully wired up)._tN)r_gateway_started_atr6r<r rr r=r=s/"D$/7!%  ! !!rcV^8dQhRR/#rFr )r s"r rrs H HD Hrc \4pVfR#VP\4\P 4\ 4\ 4\4R7R# \d\PRRR7R#i;i)z@Refresh this backend's heartbeat row. No-op when DB unavailable.N) backend_idpid started_atprofiler0z backend heartbeat refresh failedTr) rregister_backend_heartbeatr9r{r}r=r7r2r r!r")r&s r _refresh_backend_heartbeatrEso B zH %%35299;SfSh)+2B2D & F H 7$ GHsAA"BBcV^8dQhRR/#rFr )r s"r rrs Y YD Yrca\;_uu_4\'d RRR4R#RsRRR4\4\ ^8:dR#\P!4oRV3RllpV3Rlp\PV4\P!VRRR 7P4R# +'giL;i \d\P RRR7Li;i) zRegister this backend and start the refresher thread (once per process). The first refresh writes the row synchronously so this process's own sweep sees itself in the heartbeat table. ``_HEARTBEAT_REFRESH_S <= 0`` means "register once, never refresh".NTz&initial backend heartbeat write failedrcV^8dQhRR/#rFr )r s"r r8_start_backend_heartbeat_refresher..__annotate__s2242rc<SP4'g#\4SP \ 4K8R# \d\P RRR7L=i;i)z'heartbeat refresh loop iteration failedTrN)rrEr r!r"wait_HEARTBEAT_REFRESH_S) stop_eventsr _loop1_start_backend_heartbeat_refresher.._loopsW##%% W*, OO0 1 & W FQU V Ws <"A! A!c<SP4\P!\4;_uu_4\ 4;peVP \ 44RRR4R# +'giR#;ir()r rkrlr rclear_backend_heartbeatr9)r&rMs r _atexit_clear9_start_backend_heartbeat_refresher.._atexit_clearsK   + +i,**+G+IJ, + + +s (A&& A7 zhermes-gateway-heartbeat)rPrRrQ) _heartbeat_refresher_lock_heartbeat_refresher_startedrEr r!r"rLrVEventatexitregisterrWrX)rNrRrMs @r "_start_backend_heartbeat_refresherrYs # " ' '  # "(,$ #N"$q "J22K  OOM" E(B4PVVX7 # " N = MNs!B,B, B?, B< ?"C$#C$cV^8dQhRR/#rFr )r s"r rrswwwrc \^8:g\^8:g\4'gR#\;_uu_4\'d RRR4R#RsRRR4\ \\ RRR7R# +'giL);i)a Schedule the once-per-process startup orphan sweep, delayed by the WS-orphan grace window so a client reconnecting right after a restart can ``session.resume`` its row first. Grace 0 (park forever), TTL 0 and ``dashboard.startup_orphan_sweep: false`` all suppress the sweep. See #65194. NTz#startup orphan session sweep failedrZ)r)rrr_startup_orphan_sweep_lock_startup_orphan_sweep_ranrr*r rr _schedule_startup_orphan_sweepr^sc!#~':B`BbBb # # $ $  $ #%)! $!=?dluw $ #sA2A22 B cV^8dQhRR/#rFr )r s"r rrs000rc2\\4VRR7R#)zFPublish this module's helpers onto ``server``, rebound to its globals.)skipN)_)rglobals)servers&r rXrXs 6/rr()r")tuidesktopsubagentunknown)6__conditional_annotations____doc__ __future__rrksecretsrVtui_gateway._envr method_ctxrrr$r,r;r]rprsrt__annotations__rrrrrrrrrrrrrrr r#r]Lockr\rr*rTrLrUrT token_hexr8r2r9r=rErYr^rX)ris@r rrs0 #&#b:( (B6-/>.!&:6,7" 8 GI.X8*8 E& n$ VB!&^^- >3 *NPT UV$%NN,""1%C\ " H YFw$0r