+ iHjDa Rt^RIt^RIt^RIt^RIt^RIHt^RIHt^RI H t H t H t H t ^RIHt]P !R4tRtRtR tR tR t!R R 4t]!RR44tRRltRRltRRltRRltRRltR#)ueDurable cross-process session turn lease for ``TurnFacadeMixin.run_conversation``. One process at a time may load -> run -> flush a session shared through state.db (Desktop, CLI resume, gateway, background delivery). ``admit_durable_turn_lease`` acquires the row lease (or returns the early result the façade must hand back); ``DurableTurnLease`` owns the periodic refresher, the turn-liveness watchdog wiring, and the lease-loss / stall interrupt plumbing. Both timers run via the shared scheduler (``agent/periodic_scheduler.py``; timer thread orders, bodies run on per-handle workers), not per-turn threads. N) nullcontext) dataclass)AnyDictListOptional)holder_namespace_token run_agentzsession turn lease lostgr@g @g4@g@ca]tRt^"toRtRR/V3RlRlltV3RlRltV3RlR ltV3R lR ltV3R lR lt ] t RV3RlRllt V3RlRlt V3RlRlt V3RlRltV3RlRltV3RlRltV3RlRltRtRtVtR#)DurableTurnLeasea6An admitted session turn lease plus the periodic timers that keep it alive and watch the turn. ``stop`` is shared by the refresher and the liveness watchdog; ``turn_active`` gates every interrupt so a late refresher miss can never hard-interrupt the NEXT turn. Both are read and written only under ``_lock``. expires_atNc@<V^8dQhRS[RS[RS[S[,RR/#) session_idholderr returnN)strrfloat)format __classdict__s"strc  ^RIHpV!4;'g/p^RIHpVP V4wrEVeyVP VPVP4VWPPVPP4VPVPVPR7 Vn R#R# \d/pLi;i)zCreate (not schedule) the liveness watchdog when configured: lease renewal is NOT evidence of progress; a silently stalled turn would renew forever.)load_config_readonly) turn_livenessN)r timeout_spoll_s stop_event activity_lockis_turn_active commit_abortdeactivate_turn)hermes_cli.configr: Exceptionrr;resolve_turn_liveness_settingsTurnLivenessWatchdogr4r#_liveness_activity_lockr@commit_liveness_abortstop_refresherr*)r,r:liveness_configr;r<r=s& r build_threadsDurableTurnLease.build_threads>s ! >24::O ()HHY  )>> t'?'?'AY))"jj@@B#22A[A[ $ 3 3 ?DM !  ! O !sB0B00 C?Cc<V^8dQhRR/#r7r8)rrs"rrrSs @ @t @rcVP;_uu_4RVnRRR4TPPR4^RIHpV!VP4^RIHpVPPV!VPVP44VPe6VPPVPP44R#R# +'giL;i)TNzstarting new turn)arm_turn)schedule) r'r(r_touch_activity/hermes_cli.observability.shared_metrics_processrOagent.periodic_schedulerrPr+append refresh_tickr%r*)r,rOrPs& rstartDurableTurnLease.startSs ZZZ#D  ""#67L5 !!(4+<+S>S"TU == $    % %dmm&<&<&> ? %Zs C C) c<V^8dQhRR/#r7r8)rrs"rrrasrc VP;_uu_4RVnVPP4RRR4R# +'giR#;i)zStop renewal and deactivate the turn. Also the watchdog's deactivate callback: a wedge the hard interrupt cannot unwind must not keep the lease alive forever; TTL expiry lets stale-turn cleanup reclaim the row.FN)r'r(r#setr3s&rrIDurableTurnLease.stop_refresheras.ZZZ$D  IIMMOZZZs "A A c$<V^8dQhRS[RR/#)rtimeoutrNr)rrs"rrrks((E(D(rc PVPFpVPVR7K R#)zCancel both timers; ``wait=timeout`` mirrors the old ``thread.join(timeout)`` so an in-flight tick finishes before ``clear_interrupt`` runs.)waitN)r+cancel)r,r]handles&& r join_threadsDurableTurnLease.join_threadsks"((F MMwM ')rc<V^8dQhRR/#r7r8)rrs"rrrqs @ @ @rc >VPpVPPVPVP4\VRR4VP8XdRVn RVn R#R# \ d&\ PRTPRR7L]i;i)zWRelease the row and drop the agent's holder attrs (only if they still name this lease).z(Failed to release session turn lease: %sTexc_info!_active_session_turn_lease_holderN) rrrelease_session_turn_leaserrrDloggererrorr$ri&_active_session_turn_lease_ttl_seconds)r,rs& rreleaseDurableTurnLease.releaseqs  e GG . .t L 5=t D S6:E 3;?E 8 T e LLCT___cL d es0A,,-BBc <V^8dQhRS[/#r0bool)rrs"rrr|s$$$rcVP;_uu_4VPuuRRR4# +'giR#;i)N)r'r(r3s&rr@DurableTurnLease.is_turn_active|s ZZZ##ZZZs , = c$<V^8dQhRS[RR/#)rmessagerNr1)rrs"rrrs G Gs Gt Grc VP;_uu_4VPP4'gVP'g RRR4R#WnVP P VR\R7RRR4R# \d:RTP n YP n \TP n LMi;i +'giR#;i)zLease-loss interrupts fire UNCONDITIONALLY (no generation claim): a lost lease means this process no longer owns the session. Only the watchdog's stalls can be spuriously stale.NT) hard_cancel tool_reason) r'r#is_setr(r)r interrupt_REASON_LEASE_LOSTrD_interrupt_requested_interrupt_message_tool_interrupt_reason)r,rvs&&r_interrupt_turn DurableTurnLease._interrupt_turnsZZZyy!!)9)9)9Z&- " G $$W$L^$_ Z  G26 /07 -4F 1 G ZZs62C C C "BAC C  C  C  C c&<V^8dQhRS[RS[/#)rrvr)rrr)rrs"rrrsstrc VPpVP4;_uu_4\VR^4pV\VRR43VPVP38wd RRR4R#RRR4VP ;_uu_4VP P4'gVP'g RRR4R#RRR4VPVRRXR7pVRJdR#VP ;_uu_4W n RRR4R# +'giL;i +'giLf;i \d\PRRR 7RpLyi;i +'giR#;i) aCommit point for the watchdog's stall observation. Revalidates the observed ``(generation, timestamp)`` under the SAME lock ``_touch_activity`` uses, so a turn that resumed while the stall was logged is never hard-cancelled; the revalidated generation is consumed by ``interrupt(require_generation=...)`` with the first publication in ONE critical section. If ``interrupt`` raises, the abort declines FAIL-CLOSED. Returns False when stale or already winding down."_turn_liveness_activity_generation_last_activity_tsNFTzturn liveness watchdog)rxryrequire_generationz9Turn liveness abort interrupt raised; declining the abortrg)rrGr$ generation activity_tsr'r#rzr(r{rDrkdebugr))r,snapshotrvrcurrent_generation publisheds&&& rrH&DurableTurnLease.commit_liveness_aborts7   * * , ,!(0TVW!X "GE3F$MN##X%9%9S - ,- ZZZyy!!)9)9)9Z)9 T7O#5(I   ZZZ%, "-- , Z  LLT_cL dI  Zs;9D2D"8D" D5>E D " D2 5$EE E0 c<V^8dQhRR/#r7r8)rrs"rrrsBBBrc VPpV'gR#TPp^RIHp\ VRR4;'g \ 4;_uu_4\ VRR4V8wd RRR4R#RVnRVn\ VR\P!44P4RVn VPeV!RVP4RRR4R# +'giR#;i)zUClear only the interrupt admitted by this lease's refresher/watchdog. Run AFTER join.N) set_interrupt_pending_redirect_lockr~F_hard_interrupt_requested) r)rtools.interruptrr$rr}r~r!r"clear _interrupt_thread_signal_pending_execution_thread_id)r,rvr_set_interrupts& rclear_interrupt DurableTurnLease.clear_interrupts((  C U4d ; L L{} L Lu2D9WDM L*/E &'+E $ E6 8I J P P R5:E 2))5ue&@&@AM L L LsC.A$C C- c <V^8dQhRS[/#r0rq)rrs"rrrsdrc p\PRVP44VPR4R#)zRState.db stayed locked until no renewal can land before the row expires: stop now.zASession turn lease refresh stayed locked through its lifetime: %sNSession turn lease could not be refreshed; stopping to protect the transcript.F)rkwarningr4rr3s&r_stop_on_exhausted_authority-DurableTurnLease._stop_on_exhausted_authoritys-Z\`\t\t\vw mnrc  fVPP4'dR#\P!4pVP\, V, pV^8:dVP 4#VP PVP4VP\\\V4R7'dV\,VnR#VPP4'dR#\PRVP44VPR4R# \ dpTPP4'dRp?R#^RIHpT!T4'd\P!4TP&,\,TP8dTP 4uRp?#\P)RTP44Rp?R#\P)RTP4R R 7TPR 4Rp?R#Rp?ii;i) uOne periodic renewal (every ``refresh_interval`` via the shared scheduler). A real miss (rowcount 0) or a non-lock error interrupts the turn. A SQLite lock is a missed tick only while the next attempt still lands before ``_authority_deadline``. Returning False stops the timer. Acquisition reclaims a row once ``expires_at <= now``, even if the old process is alive. Holder-qualified release does not block that path, so a run of lock errors must not keep this turn active up to and past the deadline. A successful renewal is the only thing that moves the deadline. The holder-qualified UPDATE fences a late refresher from a successor lease. The façade's finally sets ``stop`` before releasing, so a holder-fenced miss observed after stop is not a loss.F) ttl_seconds patience_sNz0Lost session turn lease while turn is active: %sz>R,,. NN:DH0$H+.7H++H0) r r'rrrr)r%rr#r+r(r*)?)__name__ __module__ __qualname____firstlineno____doc__r-r4rKrVrIdeactivate_after_liveness_abortrcrnr@rrHrrrU__static_attributes____classdictcell__rs@rr r "s&ae&&"JJ* @ @'5#(( @ @$$ G GBBB$ 99rr c<a]tRtRtoRtRtRtRtV3RltRt Vt R#)TurnLeaseAdmissioniz`Outcome of ``admit_durable_turn_lease``: exactly one of ``lease`` / ``early_result`` may be set.Nc<V^8dQh/S[S[,;R&S[S[S[S[3,,;R&S[S[S[S[S[3,,,;R&#)rlease early_resultconversation_history)rr dictrrlist)rrs"rrTurnLeaseAdmission.__annotate__sQ $ %,  4S>*1  #4S#X#78? rr8) rrrrrrrr__annotate_func__rrrs@rrrsj(,E-1L;? rrcHV^8dQhR\R\R\R\/#)rrrfloorr)rr)rs"rrr s( % %C % %U %u %rc<\VRR4p\V4'd?V!W4p\V\\34'd \ V4#V\,#V\,# \ d'\ PRRR7T\,#i;i)zThe admitted row's committed ``expires_at``; when unreadable, ``floor + TTL`` (taken before acquisition, so never later than the real expiry). Overstating it lets a successor reclaim the row while this turn still believes it holds the lease.session_turn_lease_expires_atNzJCould not read the admitted turn lease expiry; using the pre-acquire floorTrg) r$callable isinstanceintrrDrkrr)rrrrreader committeds&&&& r_committed_lease_expiryr sR8$ ?F vz2I)c5\22Y''3 $ $$5$ $$ v LLeptL u $ $$ vs.A**"BBcFV^8dQhR\R\\,/#)rrr)rrrr)rs"rrrs  C HTN rc|VPV4RJ# \d\PRRR7R#i;i)zVTrue / False when the row read answered; None when it failed and the state is unknown.NzPCould not check durable session after turn lease admission; row state is unknownTrg) get_sessionrDrkr)rrs&&r_durable_session_existsrsH ~~j)55   ^  s ";;c V^8dQhR\R\R\\\3,R\\\\\3,,,R\ /#)rr relay_turn_id task_contextrr)rrrrrr)rs"rrr)sWkkk.1kAEc3hk"4S#X#78kkrc baaaa\SRR4p\VR7pVe V'gV#\SRR4'g&\\\V4RR44'gV#R\P !4 \ 4 RT R VR ,;'gR 2pR;ooR V3R llpRVVV3Rllp \P!4p VPW\\WV3RlR7'g\SWSR7Vn V#\SWQV\WQWz4R7p VSn\Sn\#WQ4p V 'dRSnS'dV RJdS'dSP'R4VP)V4p V 'd V SnWR&VP-SP*RRR7pV 'g V'd2^RIHoTP3V3RlT;'g.44WnV P74YnT# \8dT P;4hi;i)aDAcquire the session turn lease (the row need not exist yet); build (not start) its threads. Mutates ``task_context["session_id"]`` and ``agent.session_id`` when the wait forced a resume-id reload. Returns an ``early_result`` (interrupted / timed out) instead of a lease when admission fails; the caller returns it verbatim. _session_dbN)r_persist_disabledFacquire_session_turn_leasezpid=z:turn=z :platform=platformunknowncV^8dQhRR/#r7r8)rs"rr.admit_durable_turn_lease..__annotate__Gs4rc <RoR#)TNr8) reload_neededsr _on_contended/admit_durable_turn_lease.._on_contendedGs  rc(V^8dQhR\RR/#)relapsedrNr^)rs"rrrNs  % D rcj<R;ooSPVR8d R4R#R\V4 R24R#)Trug⏳ Another Hermes process is using this session; waiting for it to finish before starting your turn...u@⏳ Still waiting for the other Hermes process on this session (zs)...N) _emit_statusr)rr announcedrs&r_on_wait*admit_durable_turn_lease.._on_waitNsD$((  } D OsSZ|n\a b  rc<\SRR4#)r}F)r$)rsr*admit_durable_turn_lease..\sWU,BEJr)r wait_secondson_wait on_contended should_abort)waited)r Tz1Session is free; loading the latest transcript...r)repair_alternationinclude_row_ids"_PERSIST_AFTER_ADMISSION_INTERRUPTc3<"TFAp\V\4'gKVPS4'gK4RV9gK=VxKC R#5i)_row_idN)rrget).0mrs& r +admit_durable_turn_lease..sC  :!!!T*/0uu5W/X!*A :sA A A  A )r$rrtypeosgetpidrrrrLEASE_WAIT_SECONDS_lease_not_acquired_resultrr rrirmr_session_db_createdrresolve_resume_session_idrget_messages_as_conversationagent.session_persistencerextendrrK BaseExceptionrnr)rrrrrr admissionrrrauthority_floorrdurablelatest_session_idreloadedrrrsf$$$$ @@@radmit_durable_turn_leaser)s-  t ,B"8LMI z *E22R*FMNN ryy{m245VM? \*-::; = !&%MI  iikO ( ((9HZJ )   "\*66  T47H (Y  4 : : :  2:.  O    s05!H H%)H0HHH'"HH.c V^8dQhR\\\3,R\R\R\\,R\\,R\\\\3,,R\\,RR /#) rr user_messagepersist_user_message timestamp display_kinddisplay_metadata platform_idrN)rrrrr)rs"rrrsQQsCx.Q03QKNQQ.6smQOWX\]`be]eXfOgQ#Q QrcVPRR4pV'gVPR4'dVR9dR#^RIHp^RIHp Tp Ve/\ V\4'd\ V\4'dTp RRR WR /p \ V\4'd W8wdWR &V'dWKR &V'dW[R &VeWkR&V!VR,WR7R#)a=A follow-up that interrupted the lease wait must not consume the accepted input: append it to the early result's history so the follow-up turn sees it and persists it (the flush honours ``_PERSIST_AFTER_ADMISSION_INTERRUPT`` because this turn never owned the lease). A hard stop (``/stop``) cancels the input instead._hard_interruptedF interruptedN)append_messagerroleusercontentT api_contentr rplatform_message_idmessages)r )N) popragent.message_metadatarrrrrr) rr r r r rrhard_interruptedrrdurable_content deferred_users &&&$$$$ rcarry_unadmitted_user_messager s$''(;UC|// >>,R\B\5L"O' |T * *j9Mt.T.T. ?X\%M,$$)H'3m$(4n%,<()/:+,< +]Prc^V^8dQhR\R\R\\\3,/#)rrrr)rrrrr)rs"rrrs/22#2X\2aefiknfnao2rc R\T;'g.4R^RR/p\VRR4'd\PRV4\VRR4p\ \ \VR R44;'dVP 44pR V'dR MR /VCR R/CpV'dRVR&\VRR4'dVPVR&VP4V#Rp\PRV4VPV4R V/VCRRRRV 2RRRR/C# \dRTn RTnT#i;i \d\PRRR7LWi;i)z``waited``: another holder was seen (the user was told so), not merely an interrupt that landed before the first acquisition attempt, which no other process was involved in.r api_calls completedFr}z0session turn lease wait aborted by interrupt: %srNrzfinal_responsez[Stopped waiting for another Hermes process on this session. Your message was not processed.zAStopped before this turn started. Your message was not processed.rTrr~r)u⏳ Another Hermes process kept this session busy too long. Your message was not processed - wait for the other process to finish, then send it again.z(session turn lease wait timed out for %sz1Failed to emit session turn lease timeout warningrgfailedrlzsession_turn_lease_timeout:failure_reason session_busyfailure_retryable)rr$rkinforrrrzr~rrDr}rl _emit_warningr) rrrrbase hard_eventrresult timeout_msgs &&&$ rrrs 177R8+q+W\ ]Du,e44 F SU$?F  WZ48 9 Q Qj>O>O>Q  2T     4   *.F& ' 5. 5 5*/*B*BF& ' ,  ! ! #  P LL;ZHY K( +    $.zl;.T   ,).E &'+E $  , Y HSW XYs$ D8D<D98D9<"E! E!)rloggingrr!r contextlibr dataclassesrtypingrrrrhermes_state_pidnsr getLoggerrkr|rrrrr rrrrr rr8rrr6s  "!,,5   ; '/ ^^B @@ @ % k\QD2r