+ jHjPba0t$Rt^RIHt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI t ^RI H t ^RI Ht^RIHt^RIHt^RIHt^RIHt^R IHt^R IHt^R IHt^R IHt^R IHt]'d^RIHt^t^t^xt]P@!R]PB4t"Rt#Rt$Rt%]P@!R]PB4t&] !RR44t'RRlt(Rt)Rt*/t+R],R&/t-R],R&RRlt.R R!lt/R"R#lt0R$R%lt1]P@!R&]Pd!]4,R',]Pf4t4RR(R)llt5R*R+lt6R,R-lt7R.t8R/R0lt9R1R2lt:R3R4lt;R5R6ltR;RRllt?R;R/R?R@llt@RARBltARCRDltBRERFltCRGRRHR/RIRJlltDR;R/RKRLlltERtFRM^R;R/RNROlltGRPRQltHRRRSltI^tJ^2tKRTRUltLRVtM]P@!RW4tNRXRYltORZR[ltPRR\R]lltQ] !R^R_44tRRR`RRaR/RbRcllltSRR`RRaR/RdRellltT] !RfRg44tURRhRilltVRjRkltWRRlRmlltXRnRRoRpRqRpRrRpRsRRtRp/RuRvlltYRwRRxR/RyRzlltZR{R|lt[R}R~lt\RR/RRllt]RRRRRR/t^RRRRRpRR/RRlllt_RRlt`RRltaRRltbRRltcRRltdRRlteRRltfRRltgRth^ti^tjRRltkRRRlltlRRltmRRltnRRltoRRRlltpRRRlltqRRRRRRpRRRRRn]RM^RaRRRRRRR/ RRlltrRRltsRRlttRRltuRaR/RRlltvRRltwRRltxRRRR/RRlltyRRltzRRRRRRpRRRRRn]RM^RaRRRRRRR/ RRllt{R^/RRllt|RRlt}RRlt~RRltRRltRRltRRltRRltRRltRRlt]EP RR/RRll4tRRlt]!4tR],R&RRltRRltRRltRRltRaR/RRlltRRRRRn]RRRR/RRllt^RIHt^RIHt^RIHtR#)aDispatcher: crash/stale/orphan detection, failure accounting and the respawn circuit breaker, memory-aware concurrency caps, the one-shot ``dispatch_once`` pass, worker spawning (``_default_spawn``), worker-log rotation and the long-lived ``run_daemon`` loop. Split out of ``hermes_cli.kanban_db``; origin-resident helpers are reached late-bound via ``_kb`` (import-cycle breaking) so monkeypatching ``kanban_db.`` keeps working. ) annotationsN) dataclass)field)Path)Any)Callable)Iterable)Iterator)Mapping)Optional) TYPE_CHECKING)KANBAN_WORKER_EXIT_TRAILER)Taskz\b(quota|rate[\s_\-]?limit|429|403|auth|authenticat(?:e|es|ed|ing|ion)|authoriz(?:e|es|ed|ing|ation)|authoris(?:e|es|ed|ing|ation)|authz|unauthorized|forbidden|billing|subscription|access[\s_]denied|permission[\s_]denied|invalid[\s_]api[\s_]key)\bii,iQz-https?://github\.com/[^/\s]+/[^/\s]+/pull/\d+c]tRt^Xt$Rt^tR]R&^tR]R&]!] R7t R]R&]!] R7t R]R&]!] R7t R ]R &]!] R7t R]R &]!] R7tR]R &]!] R7tR]R &]!] R7tR]R&]!] R7tR]R&]!] R7tR]R&]!] R7tR]R&]!] R7tR]R&]!] R7tR]R&]!] R7tR]R&RtR]R&RtR]R&RtR#)DispatchResultuOutcome of a single ``dispatch`` pass. ``kanban.default_assignee`` applied this tick before spawning (#27145). Surfaces the auto-assignment to telemetry / CLI / dashboard so the operator can see when the dispatcher is acting on the fallback rule ``kanban.max_in_progress_per_profile`` (#21582). Each entry is ``(task_id, assignee, current_running_count)``. NOT an operator-actionable failure — the task will be picked up on a subsequent tick when the assignee has capacity. Separate bucket so telemetry / dashboards can show "this profile is busy" vs the board's dispatch lock (issue #35240). A losing dispatcher does no DB writes this tick — the lock holder is making progress on the same board. This is the steady-state signal that a single-writer guard is int reclaimedpromoteddefault_factory list[str]reconciled_orphansreaped_terminal_workerszlist[tuple[str, str, str]]spawnedskipped_unassignedauto_assigned_defaultskipped_nonspawnablezlist[tuple[str, str, int]]skipped_per_profile_cappedcrashed auto_blocked timed_outstalezlist[tuple[str, str]]respawn_guarded rate_limitedFboolskipped_lockedN Optional[str]memory_pressure)__name__ __module__ __qualname____firstlineno____doc__r__annotations__rrlistrrrrrrrrrr r!r"r#r%r'__static_attributes__r(B/home/ubuntu/.hermes/hermes-agent/hermes_cli/kanban_db_dispatch.pyrrXsP IsHc$)$$? ?-).t)DYD&*/*EG 'E:$)$$? ?+',T'B9BE',D&A)A9>CSW=X :XKt4GY4B#D9L)9E 6Iy6BT2E92X-24-HO*HR$D9L)9N!ND K%)O])Lr1rc V^8dQhRRRR/#)resultsz"Iterable[Optional[DispatchResult]]returnstrr()formats"r2 __annotate__r9s"Dr1cn/pRpVFpVfK VPF wrEVPV^4^,W&K" VP'd0VPR^4\VP4,VR&VP'dVPR^4^,VR&VP 'gKVP pK \ VP44UUu.uF wrgV RV 2NK pppV'dVPRV 24RPV4#uuppi)uOne line naming why the tick(s) held ready work back, or ``""``. ``active_pr=1, recent_success=2, rate_limited=1, skipped_locked=1, memory_pressure=critical`` — the respawn-guard reasons counted per task plus the tick-level holds. Feeds the "dispatcher stuck" warnings of the CLI daemon and the embedded gateway dispatcher, which otherwise report a bare zero-spawn count while ``hermes kanban tail`` is the only place the guard reason is written (#111910). Nr#r%=zmemory_pressure=, ) r"getr#lenr%r'sorteditemsappendjoin) r5countspressureres_task_idreasonkvpartss & r2describe_suppressionrKs F"H ;  # 3 3 H#ZZ2Q6FN!4    %+ZZ%BSIYIYEZ%ZF> "    '-zz2BA'F'JF# $    **H%+6<<>$: ;$:DAs!A3Z$:E ; 'z23 99U  %_record_worker_exit..s beAhr1)key) timerrLr>_RECENT_WORKER_EXITS_MAX_RECENT_WORKER_EXIT_TTL_SECONDSr@popr?) rSrTnowcutoffp_st_pidordered_s && r2_record_worker_exitris #( ))+C&)*os%;S"  #;q#@@66)=)C)C)ET)E:1grQQ)ETD $ $T4 0U  #;;-335;NO2W!23GD $ $T4 04<Us  D2D2c V^8dQhRRRR/#)r4rSrr6ztuple[str, Optional[int]]r()r8s"r2r9r9ss'@r1c\P\V44pVfR#Vwr#\V4pV^,pV^8Xd\V^, ^,4#V^8wdRV3#R#)a0``(kind, code)`` for a reaped worker PID: ``clean_exit`` (rc 0 while still ``running`` = protocol violation), ``rate_limited`` (``KANBAN_RATE_LIMIT_EXIT_CODE``, never counts as a failure), ``nonzero_exit``, ``signaled`` (``code`` is the signal), ``unknown`` (pid not in the reap registry; ``code`` None).signaled)unknownN)rLr=r_exit_code_kind)rSentryrawrh signal_numbers& r2_classify_worker_exitrrss ! $ $SX .E }  FC c(C$JMqD011M** r1c V^8dQhRRRR/#)r4coderr6ztuple[str, int]r()r8s"r2r9r9s""#"/"r1cV^8XdR#V\P8XdRV3#V\P8XdRV3#RV3#)zC``(kind, code)`` for a worker's exit code, however it was observed.r#terminal_provider nonzero_exit) clean_exit)_kbKANBAN_RATE_LIMIT_EXIT_CODE"KANBAN_TERMINAL_PROVIDER_EXIT_CODE)rts&r2rnrnsL qy   s...%% s555#T** D !!r1^z (\d+)\s*$c$V^8dQhRRRRRR/#)r4task_idr7boardr&r6 Optional[int]r()r8s"r2r9r9s! 1 13 1} 1 1r1c\P!VRVR7p\P T;'gR4pT'd\ TR,4#R# \dR#i;i)ugExit code from the trailer the worker CLI wrote to its own log; None when absent. The durable twin of ``_recent_worker_exits``: written by the worker itself (``hermes_cli.quiet_single_query.exit_single_query``), so it is there whether or not the process running this sweep ever reaped the worker. Last trailer wins — the log is append-mode across re-runs.  tail_bytesrN)rzread_worker_log Exception_EXIT_TRAILER_REfindallr)rrrpmatchess&& r2_worker_log_exit_coders]!!'d%H&&syyb1G&3wr{ 0D0 sA A%$A%cV^8dQhRR/#)r4r6z list[int]r()r8s"r2r9r9sYr1c.p\P'd{\\P 44FWwrVP 4pVfK\ V\V44\PVR4VPV4KY V#\P!R\P4wrT^8XdT#\ Y4TPT4KQ \dT#i;i \dT#i;i)aReap exited workers without blocking; returns reaped PIDs. POSIX reaps every child via ``waitpid(-1)``; Windows polls the ``Popen`` handles parked by ``_default_spawn`` (the only way to learn a child's exit code there), so the rate-limit sentinel exit is classified on both hosts.Nr)rz _IS_WINDOWSr/rMr@pollrirQr`rAoswaitpidWNOHANGChildProcessErrorr)reapedrSprocrOstatuss r2reap_worker_zombiesrs F 06689ICJ! %A*%M N  " "3 - MM#  :     jjRZZ8 ax M  , MM#  %  M    M s<C7'C%=C7C7% C40C73C44C77 DDc V^8dQhRRRR/#r4rSrr6r$r()r8s"r2r9r94s//M/d/r1c V'dV^8:dR#^RIHpV!\V44'gR#\PR8Xd{\ R\V4 R2RRR7;_uu_4pVFEpVP R 4'gKR VPR ^4^,9d R R R 4R#M R R R 4R#\PR 8Xd\P!RRRR\\V44.\P\PRRR^RR7pVP ^8wdR#R VP";'gRP%49dR#R#R# +'giR#;i \\\3dR#i;i \\P&\(3dR#i;i)uHReturn True if ``pid`` is still running on this host. Uses ``gateway.status._pid_exists`` (OpenProcess on Windows, ``os.kill(pid, 0)`` on POSIX). **DO NOT** call ``os.kill(pid, 0)`` directly on Windows — there ``sig=0`` is ``CTRL_C_EVENT`` broadcast to the console group, potentially killing unrelated processes. Zombies (exited, not yet reaped) still pass the existence check, so a worker would look "alive" forever between exit and reap. Linux: peek at ``/proc//status`` and treat ``State: Z`` as dead; macOS: ask ``ps`` for the BSD ``stat`` field and treat ``Z`` as dead. F) _pid_existslinuxz/proc/z/statusrzutf-8)encodingzState:Z:Ndarwinpsz-ozstat=z-pTreplace)stdoutstderrtextrerrorstimeoutcheckr)gateway.statusrrsysplatformopen startswithsplitFileNotFoundErrorPermissionErrorOSError subprocessrunr7PIPEDEVNULLrOrstripSubprocessError TimeoutError)rSrfliners& r2 _pid_aliver4s #(* s3x  ||w s3xj0#HHADx00$**S!"4Q"77#( IH  I6 #  ! >>tWdCCM:!!))GI D!#t{{((b//112 7IH6 )"?G<  $ ) "33\B    sa#E>*E* E*(E>2E*6E>A"F:FF* E; 5E>;E>>FF F?>F? unverifiedc V^8dQhRRRR/#)r4rSrr6r&r()r8s"r2r9r9ls 6 6c 6m 6r1c`^RIHp^RIHpV!\ V44pVfR#V!4 RV 2#)aRestart-stable identity of a live process: ``"|"``. The start time alone (``/proc//stat`` field 22 on Linux) is clock ticks since THIS boot, so a row that survives a reboot could match an unrelated process with the same PID and the same tick value; ``gateway.drain_control.current_instantiation_epoch`` (``boot_id`` + PID-1 start) changes on every reboot / container recreate, so the composed value never survives one. ``None`` when unreadable.)current_instantiation_epoch)get_process_start_timeN|)gateway.drain_controlrrrr)rSrrstarts& r2_process_fingerprintrls6 B5 "3s8 ,E })+ ,AeW 55r1c V^8dQhRRRR/#rr()r8s"r2r9r9zs . .} .T .r1cx\P!V4'gR#V\8XdR#\W4'*#)aaTrue when ``pid`` is live AND is still the worker we spawned. ``started_at`` is the fingerprint recorded by ``_set_worker_pid``; after a reboot (or any PID recycle) an unrelated process can own the number, so bare existence is never enough to extend a claim or to signal. A legacy row without a fingerprint keeps the existence answer: killing it is the pre-fingerprint behaviour and the row is rewritten with a fingerprint on its next spawn. An UNVERIFIED spawn also keeps the existence answer (a claim is never released beside a possibly-live worker) but ``_terminate_reclaimed_worker`` refuses to signal it.FT)rzrUNVERIFIED_WORKER_FINGERPRINT _pid_recycled)rS started_ats&&r2 _worker_aliverzs0 >>#  22S- --r1c V^8dQhRRRR/#rr()r8s"r2r9r9s}Tr1c*Ve V'gR#V\8XdR#\V\4'dRV9d\\ V44V8g#^RIHpHpV!\ V44pVfR#V!WA4'*# \\3dR#i;i)aTTrue when a live ``pid`` is NOT the process fingerprinted at spawn (or the fingerprint can no longer be read). Signalling it would hit a stranger. ``None`` fingerprint = legacy row, never recycled; the UNVERIFIED marker is always foreign. An integer fingerprint (rows written before the boot witness was added) compares the start time only.FTr)_start_times_agreer) r isinstancer7rrrrr TypeError ValueError)rSrrrcurrents&& r2rrs 22*c""sj'8#CH-;;I$SX.G%g::: z "s0 A==BBcV^8dQhRR/#)r4r6z$Optional[Callable[[int, int], None]]r()r8s"r2r9r9s44?4r1c^VeV#\\R4'd\P#R#)zD``signal_fn`` test hook, else ``os.kill`` when the platform has one.Nkill)hasattrrr) signal_fns&r2_kill_fnrs)b&))2773t3r1c$V^8dQhRRRRRR/#)r4rSrrrr6r$r()r8s"r2r9r9s!3MTr1c~\^ 4F-p\W4'gR#\P!R4K/ R#)z@Poll ~5 s (10 x 0.5 s) for ``pid`` to die; True once it is gone.Tg?F)rangerr]sleep)rSrrhs&& r2_poll_worker_exitrs. 2YS-- 3 r1c V^8dQhRRRR/#)r4rSrr6r$r()r8s"r2r9r9sr1cV!\V4\\R\P44R# \\ 3dR#i;i)z8Best-effort SIGKILL; True when the signal was delivered.SIGKILLTF)rgetattrsignalSIGTERMProcessLookupErrorr)rrSs&&r2_sigkillrs< SXwvy&..AB  (s/3AArrc$V^8dQhRRRRRR/#)r4rSr claim_lockr&r6zdict[str, Any]r()r8s"r2r9r9s&:: ::  :r1c RV'd \V4MRRRRRRRRR/pV'dV^8:g V'gV#\V4P\P!44'gV#RVR&\ V4pVfV#V\ 8Xd&RVR &\P!V4'*VR&V#\P!V4'd\W4'd RVR&RVR &V#RVR&V!\V4\P4\Y4'dRTR&T#\Y4'd\YP4'gT#RTR&\Y4'*TR&T# \d RTR&Tu#\dTu#i;i) uBest-effort host-local worker termination for reclaim paths. ``started_at`` is the spawn-time fingerprint: when the live process no longer matches it, the PID was recycled and nothing is signalled — the worker is gone, which is what the reclaim wanted (``terminated`` = True). An UNVERIFIED spawn (fingerprint capture failed) that is still live is never signalled either, but it is reported as surviving (``signal_refused``) so the reclaim holds the claim instead of spawning a duplicate beside it.prev_pidN host_localFtermination_attempted terminatedsigkillTsignal_refused pid_recycled)rr7rrz _host_prefixrrrrrrrrrrr)rSrrrinfors&&$$ r2_terminate_reclaimed_workerrs CHee5 D #(* z? % %c&6&6&8 9 9 D I D | 22!% !$!44\  ~~c}S==!\#^ $(D ! SXv~~&))!\ S%%""KY*3;;D K! "\  s3 E))F =F F  F c V^8dQhRRRR/#r4connsqlite3.Connectionr6rr()r8s"r2r9r9s 2)r1c xVPR\\P!44\, 34P 4p\ P !4p.pVFp\WW1V4K V# \d6\ PPRTR,TR,RR7KVi;i)uEnd host-local workers that outlived their run (issue #111791) — a worker that called ``kanban_complete`` and then hung keeps its ``state.db`` sidecar fds open and no ``running``-only sweep can see it once ``tasks.worker_pid`` is cleared. Keys on the closed ``task_runs`` row's retained pid + spawn fingerprint: a legacy row (NULL fingerprint) or a recycled PID is never signalled; a pid that is simply gone just has its evidence cleared. A run that ended less than ``TERMINAL_WORKER_REAP_GRACE_SECONDS`` ago is left alone so a worker still finalising after its own transition is not killed. One row's failure (signal, /proc probe) is logged and skips only that row. Returns the task ids whose worker was terminated.zSELECT id, task_id, worker_pid, worker_started_at, claim_lock FROM task_runs WHERE ended_at IS NOT NULL AND ended_at <= ? AND worker_pid IS NOT NULL AND worker_started_at IS NOT NULLzAkanban dispatch: terminal worker reap failed for run %s (task %s)idrTexc_info) executerr]"TERMINAL_WORKER_REAP_GRACE_SECONDSfetchallrzr_reap_terminal_worker_rowr_logdebug)rrrows host_prefixrrows&$ r2reap_terminal_workersrs << G TYY[ > >@   hj  ""$KF  %d P M   HHNNSD 3y>D   s& A990+MMVYZ^V_    c)n%   s )AE'' E7 c V^8dQhRRRR/#)r4rdictr6r$r()r8s"r2r9r9/s  d t r1c\VPR4;'dIVPR4;'gVPR4;'dVPR4'*4#)uTrue when we tried to kill our own host-local worker and it is still alive. Reclaiming then would release the claim and spawn a second worker while the first still runs — the duplication loop. Only host-local workers we actually signalled count; a non-local lock or no-op attempt (no ``os.kill``) must fall through to the normal release path since we cannot manage that worker anyway. rrrr)r$r=)rs&r2_worker_survived_terminationr/s_  % . . __4 5 Z ZIY9Z . . - - r1c4V^8dQhRRRRRRRRR R R RR R /#)r4rrrr7rr&rarrrrGr6rUr()r8s"r2r9r9>sWUU U UU U  U U Ur1c V\P,p\P!V4;_uu_4VPRWaV34pVP^8wd RRR4R#\P !W4pVeVPRWh34RVRVRV/p V P V4\P!WRWR7RRR4R# +'giR#;i) aCHold a claim whose worker survived termination instead of releasing it. Extends ``claim_expires`` by ``RECLAIM_DEFER_GRACE_SECONDS`` so the task stays ``running`` (no duplicate spawn) and records ``reclaim_deferred``. The next tick retries the kill; not spawning a duplicate is what lets the throttled worker finally die. zZUPDATE tasks SET claim_expires = ? WHERE id = ? AND status = 'running' AND claim_lock IS ?Nz3UPDATE task_runs SET claim_expires = ? WHERE id = ?rGrclaim_expires_nowreclaim_deferredr )rzRECLAIM_DEFER_GRACE_SECONDSrrrowcount_current_run_idupdater) rrrrarrGgracecurrpayloads &&&&&$ r2_defer_reclaim_for_live_workerr$>s #11 1E t  ll F Z (  <<1    $$T3   LLNQVP_ `V\:?RTYZ{# $);WT    s'C'AC C! noteexpected_run_idc ,V^8dQhRRRRRRRRR R /#) r4rrrr7r%r&r&rr6r$r()r8s"r2r9r9_s:## # #  # # #  #r1c  \\P!44p\P!V4;_uu_4RpWA3pVeVR, pV\V43, pVP WV4pVP ^8wd RRR4R#Ve \V4M\P !W4pVeVP RWH34\P!YRV'dRV/MRVR7RRR4R # +'giR #;i) aRecord a ``heartbeat`` event + touch ``last_heartbeat_at``. Liveness signal orthogonal to the PID check: a worker whose forked child (train loop, crawl) is stuck can still have a live Python process. Returns False if the task is not running or its claim expired. zJUPDATE tasks SET last_heartbeat_at = ? WHERE id = ? AND status = 'running'Nz AND current_run_id = ?Fz7UPDATE task_runs SET last_heartbeat_at = ? WHERE id = ? heartbeatr%r T)rr]rzrrrrr) rrr%r&rasqlparamsr"rs &&$$ r2heartbeat_workerr,_s diik C t  Z  & , ,C s?+- -Fll3' <<1   *  $$T3    LLRUXTa b  ;"VTN ! * +  * sA C<A"C<< D c V^8dQhRRRR/#rr()r8s"r2r9r9scc0c cr1c.p\\P!44p\P!4pVP R4P 4p^RIHpHpVEFpV!WR,4'dKVR,;'gRp V PV4'gKGT\P!VR,4;'gT, p \VR,4p W8dK\VR,4p VR,p \P!VR 4pV\8Xd@\P!V 4'd$\PPR W4EK R p\!V4pVe\P!V 4'd\#W4'gp\$P&!\(\*4;_uu_4V!V \,P.4R R R 4\1W4\3W4'd \5VV 4pR \V 4 RV R2p\P6!V4;_uu_4V!W RR7'd R R R 4EK\P8!W 4pVP RVWVR,34pVP:^8Xd\RV R\V 4RV RVRV/p\P<!W RRVVR7p\P>!W RVVR7VPAV 4R R R 4XP:^8XgEK\CW VRR R RV RVRX/R7EK V# +'giEL^;i +'giLV;i)uyTerminate workers whose per-task ``max_runtime_seconds`` has elapsed. SIGTERM, short grace, then SIGKILL. Emits ``timed_out`` and restores the task's source phase so the next tick re-spawns the same kind of worker — unless the circuit breaker already gave up, leaving it blocked. Host-local only (same reasoning as ``detect_crashed_workers``). ``signal_fn`` is a test hook. aqSELECT t.id, t.worker_pid, t.worker_started_at, COALESCE(r.started_at, t.started_at) AS active_started_at, t.max_runtime_seconds, t.claim_lock FROM tasks t LEFT JOIN task_runs r ON r.id = t.current_run_id WHERE t.status = 'running' AND t.max_runtime_seconds IS NOT NULL AND COALESCE(r.started_at, t.started_at) IS NOT NULL AND t.worker_pid IS NOT NULLowner_reclaim_blocked owner_reclaim_blocked_after_killrrractive_started_atmax_runtime_secondsr r z^kanban: task %s worker pid %s exceeded max runtime but has no verified identity; not signalledFNzelapsed z s > limit sT signalledzUPDATE tasks SET status = ?, claim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL, last_heartbeat_at = NULL WHERE id = ? AND status = 'running' AND worker_pid = ? AND claim_lock IS ?rSelapsed_seconds limit_secondsr retry_statusr outcomererrormetadatar )r<r; release_claimend_runevent_payload_extra)"rr]rzrrr hermes_cli.kanban_owner_recoveryr0r1r _to_epoch_row_getrrrwarningrr contextlibsuppressrrrrrrrr_retry_status_for_runr_end_runrrA_record_task_failure)rrr rarrr0r1rlockelapsedlimitrStidrkilledrr<r9r"r#rs&$ r2enforce_max_runtimerOsI diik C""$K << ) hj i 4y 1 1 < &&B{++ s+>'?@GGCHC-./ ? #l#$$i\\#':; 6 63>>#;N;N HH  78; B  "  S^^C%8%8]3=[=[$$%7AAS&..)B c .S--!$,3w<. 5'; ]]4 0TJJ ! 44T?L,,; s\):; C||q 3%s7|#Uv"L {;'!!$['&Q  %7!> <<1  ##%*CFNT`$a   IX aBAA! sL<M$BM< M  M stale_timeout_secondsc$V^8dQhRRRRRR/#)r4rrrPrr6rr()r8s"r2r9r9s-dd dd  dr1cV^8:d.#\\P!44p.pVPR4P4p^RIHpHpVEFpV!WR,4'dKVR,fK(T\P!VR,4;'gT, p W8dK^\P!VR,4p V e W:, MRp V eV \8dKVR,p VR,p VR,;'gR p\P!WV\P!VR 4R 7p\V4'd\W WVR R 7EK\P!V4;_uu_4V!W \VP!R44R7'd RRR4EKc\P"!W 4pVPRVWR,34pVP$^8wd RRR4EKR\V 4R\P&!V 4R\P&!V 4RTRV 'd \V 4MRRV/pVP)V4\P*!Y RRV eR\V 4 R2MRR\V 4 R2,VR7p\P,!W RVVR7VP/V 4RRR4EK V# +'giEK;i)aIReclaim ``running`` tasks with no heartbeat progress; returns their ids. Stale = running longer than ``stale_timeout_seconds`` (active run's ``started_at``, else ``tasks.started_at``) AND ``last_heartbeat_at`` NULL or older than ``_STALE_HEARTBEAT_GAP_SECONDS``. Task returns to its source phase, run closes ``outcome='stale'``, a live host-local worker is killed. ``0`` disables the check; ``signal_fn`` is a test hook. Deliberately NOT counted via ``_record_task_failure``: an absent heartbeat is not a worker failure, and counting it would let long-running tasks trip the breaker. zSELECT t.id, t.worker_pid, t.worker_started_at, t.last_heartbeat_at, t.claim_lock, COALESCE(r.started_at, t.started_at) AS active_started_at FROM tasks t LEFT JOIN task_runs r ON r.id = t.current_run_id WHERE t.status = 'running'r/rr2Nlast_heartbeat_atr rrr r heartbeat_stale_worker_alive)rGrr5zUPDATE tasks SET status = ?, claim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL, last_heartbeat_at = NULL WHERE id = ? AND status = 'running' AND claim_lock IS ?r7heartbeat_age_secondstimeout_secondsrSr9r!zno heartbeat for zs zno heartbeat everz after z s runningr:r )rr]rrrAr0r1rzrB_STALE_HEARTBEAT_GAP_SECONDSrrCrr$rr$r=rGr_opt_intr rHrrA)rrPrrarrr0r1rrKlast_hbhb_agerSrMrJrr9r"r#rs&$$ r2detect_stale_runningr[s ! diik CI << %  hj  i 4y 1 1  " # + s+>'?@GGCH  * --$7 89$+$7#-T  &+G"G ,$i< &&B55 s||CI\7]_ ( 4 4 *4k5   ]]4 /T+//ZqJrEstt! 44T?L,,( s $56 C||q ! ""3w<#S\\'%:'f)=!#83s3xD G NN; '\\)(F }B7,c'l^95 6 ! F   d'& I   S !K! =J O! s*K#6AK#CK## K5 c V^8dQhRRRR/#rr()r8s"r2r9r9WsDD%7DIDr1c J\\P!44p.pVPR4P4p^RIHpVEFpV!WR,4'dKVR,pVR,pV'dJ\ V\P!VR44'd#\PPRWg4K~\P!V4;_uu_4V!W4'd RRR4KVPRWeR ,VR ,34pVP^8wd RRR4KR R R VR ,R \P!VR ,4RV'd \V4MRR V/p \P!WRRRV R7p \P!WRRV4\P !WRWR7VP#V4RRR4\PP%RWeR ,V4EK V# +'giL?;i)aRequeue ``running`` cards with broken claim bookkeeping; returns their ids. A task ``running`` with NULL ``claim_lock``/``claim_expires`` (crash mid-claim, manual SQL, DB restore) is a zombie forever: ``release_stale_claims`` needs ``claim_expires``, ``detect_crashed_workers`` needs a host-local lock + pid, ``detect_stale_running`` is off by default. Orphans go back to ``ready`` with a comment, leaked run closed, ``reconciled`` event; a row with a live host-local PID is deferred so no duplicate spawns beside it. zSELECT id, claim_lock, claim_expires, worker_pid, worker_started_at FROM tasks WHERE status = 'running' AND (claim_lock IS NULL OR claim_expires IS NULL))r0rr r uekanban reconcile: task %s has broken claim bookkeeping but pid %s is alive on this host — deferringNzUPDATE tasks SET status = 'ready', claim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL, last_heartbeat_at = NULL WHERE id = ? AND status = 'running' AND claim_lock IS ? AND claim_expires IS ?r claim_expiresrGorphaned_runningrarz0orphaned running card (broken claim bookkeeping)r: dispatcheru_reconciliation: card was 'running' with no valid claim (dead/gone worker) — requeued to ready reconciledr zRkanban reconcile: requeued orphaned running task %s (claim_lock=%r, worker_pid=%r))rr]rrrAr0rrzrCrrrrrBrH_insert_commentrrAr) rrararr0rrMrSr"r#rs & r2reconcile_orphaned_runningrcWs diik CJ << > hj  G 4y 1 1 $i, =cll38K&LMM HHNN=>A   ]]4 $T//! ,,? ,'_)=> C||q ! ,c,/s?/C!D#c#h4s G\\#KH F   <;     dw N   c "C!D   -.1|3Dc ]d M! s$H>4H<0H-A.H H" c V^8dQhRRRR/#)r4 error_textr7r6r()r8s"r2r9r9s33r1c\P!RRVR,4p\P!RRV4pVP4P4#)zTNormalize an error message (strip PIDs, timestamps) so same-root-cause errors group.z \bpid \d+\bzpid N:NPNz \b\d{10,}\bz)resublowerr)refps& r2_error_fingerprintrls? C 9B  +B 88:   r1c$V^8dQhRRRRRR/#)r4rrrr7r6rr()r8s"r2r9r9s"%7##r1cZ^pVPRV\34P4pVF|pVR,;'gRpVR8XdKVR8XdV\P!VR,4P R4'gRVR ,;'gR9d V^, pK{V# V#) aCount the task's trailing run of clean-exit protocol violations. Walks closed runs newest-first (including the one ``detect_crashed_workers`` just closed). ``rate_limited`` runs are neutral and skipped (a quota wall says nothing about the task); any other closed run breaks the streak, so the budget counts ONLY protocol violations. Violations are recognized by the ``protocol_violation`` run-metadata marker, with the error text as fallback for runs recorded before the marker existed. zrSELECT outcome, error, metadata FROM task_runs WHERE task_id = ? AND ended_at IS NOT NULL ORDER BY id DESC LIMIT ?r;rr#rr=protocol_violationzprotocol violationr<)r_PROTOCOL_VIOLATION_SCAN_LIMITrrz _json_dictr=)rrstreakrrr;s&& r2_protocol_violation_streakrssF << # 01   hj  i.&&B n $  i  NN3z? + / /0D E E#G (:(:; aKF  M Mr1u/worker exited cleanly (rc=0) without kanban_complete, kanban_block or kanban_request_review — protocol violation. If the prior run already did the work, verify it and report it via kanban_complete (or kanban_request_review); a run without a terminal kanban call counts as failed no matter what it did.u7[─━═╭╮╰╯│┃┌┐└┘]+|☤\s*HermescV^8dQhRR/#r4r6r7r()r8s"r2r9r9s--c-r1c^RIHpV!R4#)z^The CLI exit-summary header (``cli_session_mixin.show_exit_summary``), in the active language.rezcli.session.exit_resume_hint agent.i18nrerws r2_exit_summary_markerrzs + ,,r1cV^8dQhRR/#)r4r6ztuple[str, ...]r()r8s"r2r9r9sGG_Gr1c$^RIHpRRV!R43#)ryrwz session_id:zQuery:zcli.chat.initializing_agentrxrws r2_log_noise_prefixesr}s 8Q'D%E FFr1c$V^8dQhRRRRRR/#)r4rr7rr&r6r()r8s"r2r9r9s!""#"m"s"r1c\P!VRVR7pT'gR#\P RT4pTP \ 44pTR8wdTRTp.pTP4Fcp\P RT4P4pT'gK1TP\44'dKRTPT4Ke RPT4RR# \dR#i;i)aBest-effort read of a dead worker's last printed text, for the board diagnostic. A ``chat -q`` worker's stdout/stderr are redirected to its per-task log (``_default_spawn``), so when it exits without a terminal board call the reason is usually sitting there: the model's own explanation of why it could not comply (#88603), or the rendered provider error (#46593). The reap used to discard it in favour of a canned message on every retry. Trims the CLI exit summary, rule lines and the ``session_id:`` trailer; returns "" (never raises) on a missing/empty log. ``board`` must come from the dispatching tick: ambient current-board resolution is wrong for every board but the one the dispatcher thread happens to call "current", so the log would silently not be found. rrrN rip)rzrrrrirfindrz splitlines _LOG_CHROMErrr}rArB)rrrpcutlineslns&& r2_worker_final_outputrs!!'d%H    r3 'C ))(* +C by$3i Enn __R $ * * , 2bmm$7$9:: LL  88E?45 !! sC-- C<;C<c]tRtRt$RtR]R&R]R&R]R&R]R&R ]R &R tR ]R &R tR ]R&R tR ]R&] RRl4t Rt R#) _DeadWorkeriz;How ``detect_crashed_workers`` should book one dead worker.r7kindrrtre event_kindr event_payloadFr$ror#rvcV^8dQhRR/#rur()r8s"r2r9_DeadWorker.__annotate__ sBBSBr1c .VP'dR#R#)r#rr#)selfs&r2 run_outcome_DeadWorker.run_outcomes"&!2!2!2~A Ar1r(N) r)r*r+r,r-r.ror#rvpropertyrr0r(r1r2rrs\E I OO$$L$#t#HBBr1rrrc ,V^8dQhRRRRRRRRRR/# r4rSrclaimerr&rrr6rr()r8s"r2r9r9&s6 $BO r1c\WW#VR7pV'dRVP'g@\W4R7pV'd,V;PRV: 2, unWePR&V#)a2Map a dead worker's reaped exit status to its reclaim bookkeeping. A clean exit or a crash carries the worker's own last output (``worker_output`` in the event payload, appended to the error text) so the board and the retry worker see WHY instead of a bare label; a rate-limited requeue does not need it. rrrz Worker's last output: worker_output)_classify_dead_worker_exitr#rrer)rSr exit_coderrdeadrs&&&$$ r2_classify_dead_workerr&sX &cIV[ \Dt(((,WB  OO!88IJ JO2?   / Kr1c ,V^8dQhRRRRRRRRRR/#rr()r8s"r2r9r99sHDIDI DI DI  DI  DIDIr1c\V4wrVVe\\V44wrVM,VR8Xd&V'd\W4R7pVe\V4wrVVR8Xd\ WV\ RRVRVRVRR/RR 7#VR 8Xd\ WVR V R 2R RVRVRV/RR 7#VR8Xd"\ WVR V RV R2RRVRVRVRVRR/RR7#VR8Xd R V RV 2pMVR8Xd R V RV 2pMR V R2pRVRV/p VeVR8wd WYR&WiR&\ WVVRV 4#)amExit status -> reclaim bookkeeping, before the worker's own words are folded in. ``exit_code`` is the authority's exact-run result for a managed worker (a child of the profile owner, never of this dispatcher, so it cannot be reaped here). Otherwise the reap registry only knows children of THIS process; a per-tick dispatcher reads the exit trailer the worker left in its log instead, so the same death gets the same booking (protocol violation / rate-limit requeue / crash) as under the gateway-embedded dispatcher. A worker that never reached its exit epilogue (killed, OOM) leaves no trailer and stays a plain crash. rmrrxrorSrrT)ror#zpid uI exited rate-limited (quota wall) — requeued without counting a failurerrvz+ exited on a terminal provider error (exit ud): the provider rejected this profile's credential or model — fix the configuration, then unblock.r exit_kind)rvrwz exited with code rlz killed by signal z not alive)rrrnrrr_PROTOCOL_VIOLATION_ERROR) rSrrrrrrtloggedrers &&&$$ r2rr9s"'s+JD$S^4 d  w&w<  (0JD | 13GCG[$@TVZ [#    ~ 3%` a  CG[$ ?    "" 3%B4&IZ Z  CG[$ TSfhl m"    ~C5 24&9  C5 24&9 C5 + CG4M DI-%)k"%)k" t:y- HHr1c]tRtRt$Rt]!]R7tR]R&]!]R7t R]R&]!]R7t R]R&]!]R7t R ]R &R t R #) _CrashSweepizFEverything ``detect_crashed_workers`` collects inside its reclaim txn.rrrr#z'list[tuple[str, int, str, _DeadWorker]] crash_detailsz list[dict]exited_hook_payloadsr(N) r)r*r+r,r-rr/rr.r#rrr0r(r1r2rrsIPt4GY4#D9L)9>CSW=XM:X(-T'B*Br1rc$V^8dQhRRRRRR/#)r4rrrr&r6rr()r8s"r2r9r9s(JJ 2J=JT_Jr1c\4p\P!V4;_uu_4VPR4P 4p\P !4p^RIHpHpVEF8pVR,;'gRpVPV4'gK0\P!\P!VR44p V e7\P!4V , \P!48dK\VR,\P!VR44'dKV!WR ,R R 7'dK\VR,4p V!WR ,WR,4p \!WR,WR ,VR 7p \P"!WR ,4p WP$R &VPRWR ,WR,34pVP&^8wdEK\P(!WR ,V P*V P*V P,\/V P$4R7p\P0!WR ,V P2V P$VR7VP4P7RVR ,RVR,RVRV RV P8RV P:RV P*R V /4V P<'gV P>'d-VPRV P,R,VR ,34V P<'d&VP<P7VR ,4EKVP@P7VR ,4VPBP7VR ,WR,V 34EK; RRR4V# +'giT#;i)zCRelease every host-local ``running`` task whose worker PID is dead.zSELECT id, worker_pid, worker_started_at, claim_lock, started_at, assignee FROM tasks WHERE status = 'running' AND worker_pid IS NOT NULL)managed_run_exit_coder0rrrNr r rT) crash_sweeprr9zUPDATE tasks SET status = ?, claim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL WHERE id = ? AND status = 'running' AND worker_pid = ? AND claim_lock IS ?r:r rassigneerrrr;z4UPDATE tasks SET last_failure_error = ? WHERE id = ?NiN)"rrzrrrrrArr0rrBrCr]_resolve_crash_grace_secondsrrrrGrrrHrrerrrrrArrtr#rorr)rrsweeprrrr0rrJrrSrrr9r"rs&& r2_reclaim_dead_workersrs ME t  || B  (* &&( aC|$**D??;//s||C'FGJ%$))+ *BSEeEeEg*gS. SBU0VWW%Tt9$GGc,'(C-dIs DUVI(,.?X\T]ejkD44Tt9EL1=  ~ .,,;4y# V#)uCount each crash against the breaker; returns the task ids it tripped. Protocol violations get a BOUNDED violation-only budget independent of ``consecutive_failures`` (per-task ``max_retries`` takes precedence); systemic same-error crashes (>= 3 identical fingerprints this tick) and terminal provider errors (credential revoked, model gone — a retry cannot heal them) trip immediately. z*SELECT max_retries FROM tasks WHERE id = ?N max_retriesrTFrSrprotocol_violationsprotocol_violation_limit)r<r; failure_limit force_tripr>r?r@rv)r<r;rr>r?r@sticky)r<r;rr>r?r@)rlrer=rorsrfetchonerzrCr!_PROTOCOL_VIOLATION_FAILURE_LIMITrIrvrA)rrr fp_countsrhrrkrMrSrrerrtrow task_overrideviolation_limittripped is_systemicextras&& r2_account_crashesrs!L "I& 1a  0! b!,q0 '$1'__  " " "/:F<< LsfU^^`D|LL}=M&3&?M"Ef '+ !-#3w)6. %G # # # + !#%*CGEXZ^$_G$--(::(FJaOKCG4E"&h* !#.aD#$)G 7    $}$1~ r1c$V^8dQhRRRRRR/#)r4rrrr&r6rr()r8s"r2r9r9+s#..!3.M.U^.r1c\WR7pVP'd\WP4M.pV\nVP \nVP'dy\P!R4'd]\P!4pVPF7p\V4p\P!RVPR43RV/VBK9 VP#)uReclaim ``running`` tasks whose worker PID is no longer alive. Restores the source phase immediately (no waiting for the claim TTL), for tasks claimed by *this host* only — other hosts' PIDs are meaningless. Clean exit while ``running`` is a protocol violation with a bounded violation-only retry budget; ``KANBAN_RATE_LIMIT_EXIT_CODE`` is a quota wall, released WITHOUT counting a failure and surfaced via the ``_last_rate_limited`` attribute (the return stays crashed-only). ron_kanban_worker_exitedrr)rrrdetect_crashed_workers_last_auto_blockedr#_last_rate_limitedrrz_kanban_observer_consumedget_current_boardr_fire_kanban_lifecycle_hookr`r)rrrr_board hook_fieldss&& r2rr+s "$ 4EBGBUBUBU#D*=*=>[]L1=-050B0B- !!!c&C&CD]&^&^&&( 55K{+K  + +"* *% &' () 60 ==r1rrFr>r?r@infrastructurecDV^8dQhRRRRRRRRRRR R R R R R R RRR RR / #)r4rrrr7r<r;rz int | Nonerr$r>r?r@zOptional[dict]rr6r()r8s"r2r9r9\sxx x x x  x  xxxx(xx xr1cVf\pVR,p\P!V4;_uu_4VPRV34P 4p V f RRR4R#V'd\P !WV R,4MV R,R8XdRMRp \ V R ,4V 'd^M^,p \P!V R 4p V e\ V 4R rM \ V4R rV 'gV'gW8gV'dVPR WW!34MVPRWV34V'dLRV RV /pV 'dRVR&\P!WW3VVR7p\P!WVRV/VCVR7RRR4R#TPRV'dRMR,R,WV34RV RVRVRVRVRV /pRpV'd&\P!WRRVRV RVRVRVRV /R7pV'dRVR&V'dVPV4\P!WRVVR7RRR4R# +'giR#;i)uRecord a non-success outcome and maybe trip the circuit breaker; every non-success path funnels through here so ``consecutive_failures`` stays consistent. Returns True when the task was auto-blocked. ``release_claim=True, end_run=True``: spawn-failure path (task still running with an open run — restore source phase or ``blocked``, release claim, close run). Both False: timeout/crash path (caller already restored the phase and closed the run; only the counter moves, a trip flips to ``blocked`` + ``gave_up``). Threshold: per-task ``max_retries`` > ``failure_limit`` > ``DEFAULT_FAILURE_LIMIT``. ``force_trip`` trips unconditionally (caller applied its own bounded-retry policy). ``infrastructure=True``: the host refused the spawn (no restart-safe scope, #114720) — nothing about the card ran, so the run and event are recorded with ``infrastructure: true`` but ``consecutive_failures`` is left alone and the breaker never trips; the card stays retryable and :func:`check_respawn_guard` spaces the retries. NrzXSELECT consecutive_failures, status, max_retries, current_run_id FROM tasks WHERE id = ?Fcurrent_run_idrreviewreadyconsecutive_failuresrtaskr`zUPDATE tasks SET status = ?, claim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL, consecutive_failures = ?, last_failure_error = ? WHERE id = ? AND status = 'running'zNUPDATE tasks SET consecutive_failures = ?, last_failure_error = ? WHERE id = ?failuresr9Trr:r<r z%UPDATE tasks SET status = 'blocked', zVclaim_lock = NULL, claim_expires = NULL, worker_pid = NULL, worker_started_at = NULL, rzjconsecutive_failures = ?, last_failure_error = ? WHERE id = ? AND status IN ('running', 'ready', 'review')effective_limit limit_sourcetrigger_outcomegave_upr) DEFAULT_FAILURE_LIMITrzrrrrGrrCrHrr )rrr<r;rrr>r?r@rrr9rrrrdetailrr#s&&&$$$$$$$ r2rIrI\sk>- $KE t  ll &(/z  (*  ;    % %dS9I5J K!(mx7(W  s123NqPQR S-8  $,/ ,>\,/ ,> \ *0K :"U<  :g. $h M!/3F+,7%Z`!!$7E:TV:T]cd]  d 3h%' )H H g &     L U w L   \\y%%w%"L"L  F !%GH   NN. / $GFKm    sO'I #I +BI :I  I 0I  I >I I +AI  I ,I  I rrc(V^8dQhRRRRRRRR/#) r4rrrr7rSrr6rUr()r8s"r2r9r9s0pp,psppjnpr1c \\V44;'g\p\P!V4;_uu_4Vf\P !W4pVP R\V4WQW4V34pVP'dVP R\V4WS34\P!WRR\V4RV/VR7RRR4R# +'giR#;i)aPublish the launcher only while this claim has no executing worker yet, together with its restart-stable fingerprint (``_process_fingerprint``); the ``spawned`` event carries both. The fingerprint is what lets every later liveness/kill decision tell OUR worker from a process that recycled the PID after a reboot. A failed capture is persisted as ``UNVERIFIED_WORKER_FINGERPRINT``, never NULL: NULL is the legacy pre-fingerprint row whose bare-PID kill authority a new spawn must not inherit.NzUPDATE tasks SET worker_pid = ?, worker_started_at = ? WHERE id = ? AND status = 'running' AND current_run_id IS ? AND worker_pid IS NULL AND (? IS NULL OR claim_lock = ?)GUPDATE task_runs SET worker_pid = ?, worker_started_at = ? WHERE id = ?rrSrr ) rrrrzrrrrr)rrrSrrrr"s&&&$$ r2_set_worker_pidrs&c#h/PP3PJ t   >((7Fll 0XzF K M <<< LLbc(J7 9 $UCHlT^4_hno    s BC C, c ,V^8dQhRRRRRRRRRR /#) r4rrrr7rrrSr6r$r()r8s"r2r9r9s1-SsW[r1c \\V44;'g\p\P!V4;_uu_4VP RV34P 4pVe&VR,R8wgVR,\V48wd RRR4R#VR,fVR,;'gR P\P!44'duVP R \V4WA34VP R \V4V\V434\P!WR R \V4RV/\V4R7RRR4R# +'giR#;i)aWorker-side half of ``_set_worker_pid``, run by the worker before its first model call. A dispatcher killed between spawning the worker and ``_set_worker_pid`` leaves the run with no pid: no liveness check can see the worker, so a TTL expiry reclaims the card and spawns a second worker beside it. The worker fills the missing pid itself (``worker_registered``). False when ``run_id`` is no longer the card's live run: the card was reclaimed before this worker got here, and it must exit without working it.zMSELECT status, current_run_id, worker_pid, claim_lock FROM tasks WHERE id = ?NrrunningrFr rrzCUPDATE tasks SET worker_pid = ?, worker_started_at = ? WHERE id = ?rworker_registeredrSrr T) rrrrzrrrrrr)rrrrSrrs&&&& r2adopt_worker_pidrs+&c#h/PP3PJ t  llj#:''/xz ;#h-94   d-@5#c(T`blBm%([ 2     s A EE2*EA4E E, c$V^8dQhRRRRRR/#)r4rrrr7r6rUr()r8s"r2r9r9s"   !3  c  d  r1c\P!V4;_uu_4VPRV34RRR4R# +'giR#;i)zReset the unified consecutive-failures counter. Called from ``complete_task`` on success. NOT called on spawn success: a spawn proves the worker could start, not that the run will succeed, so timeouts and crashes must accumulate across spawn boundaries. zQUPDATE tasks SET consecutive_failures = 0, last_failure_error = NULL WHERE id = ?N)rzrr)rrs&&r2_clear_failure_counterrs8 t    5 J     s = A lanerc(V^8dQhRRRRRRRR/#)r4rrrr7rr6r&r()r8s"r2r9r9s2ee e'*e58eer1c VPRV34P4pVfR#\\P!44p\P !4pVPRV34P4pVexVR,R8XdjV^8dc\P !VR,4PR4'd1\P!VR,4pVeWG, V8dR #VeJVR,R 8Xd<V^8:dR#\P!VR,4pVeWG, V8dR #R#\P!VR ,4pVe VR,MRp V'd%V R 8wd\PV4'dR#VR8XdR#\R\VRW\, R44RR7p V e \\VRWR44'gR#\VRW\ , RVR7U U u.uFIwr\"P\P!V R,4;'gR4'gKGV NKK p p p V 'dq\;QJd5R\VRV\V 4RRRR 74F 'gK RM* R!M&!R\VRV\V 4RRRR 744'dR#R"#R#uup p i)#uiReturn a guard reason if ``task_id`` should NOT be re-spawned, else None. Called per ready/review row before any claim attempt. Priority order: ``"infrastructure_cooldown"`` (latest run is a ``spawn_failed`` the host refused — no restart-safe scope — within the cooldown; never counted), ``"rate_limit_cooldown"`` (latest run ``rate_limited`` within the cooldown; checked BEFORE ``blocker_auth`` because the requeue stamps a quota-flavored ``last_failure_error`` that would otherwise park the task forever — that path never increments ``consecutive_failures``), ``"blocker_auth"`` (quota/auth pattern; the breaker still trips eventually), then for the ready lane only ``"recent_success"`` (completed run within the window, unless a re-queue event arrived after it — a deliberate re-run) and ``"active_pr"`` (PR URL in a recent comment; re-spawning risks a duplicate PR — unless a handoff event followed the comment: the named profile must work on that PR). The review lane skips the last two: they are the *inputs* to a review handoff. Stale / dead claim locks are NOT a guard reason — the reclaim passes own those. z1SELECT last_failure_error FROM tasks WHERE id = ?Nz{SELECT outcome, ended_at, metadata FROM task_runs WHERE task_id = ? AND ended_at IS NOT NULL ORDER BY ended_at DESC LIMIT 1r; spawn_failedr=rended_atinfrastructure_cooldownr#rate_limit_cooldownlast_failure_errorr blocker_authrc3*"TF wrVxK R#5iNr().0rerhs& r2 &check_respawn_guard..csg"f1"fs task_runszAND outcome = 'completed')default task_eventsz"I ?s~~j.DEIIJZ[[}}Z %;"I ! ==J!78  S^{$B( //#23 4C.8.DZ *$N ~*/B/I/I#/N/N x g k7*G$GId"fgptvG3{ -3q(s$t$t*$PhJh/7CIVIda+223??1V93M3S3SQSTIHV 3zkmWc(mRYjswGyz333zkmWc(mRYjswGyz z z Vs9K KKrrr created_atrrrrrc<V^8dQhRRRRRRRRRRR RR R R R RR/ #)r4rrtabler7rsincerwhererrr$rrr6zIterator[tuple[int, Any]]r()r8s"r2r9r9sQ  (  s 3 WZ  (, BO [t r1c #*"\V,pRT T RT RT RT RV'dRMR R2 p VPWV34FLp \P!V ^,4p V fTMT p V fK.V'd W8gK>MW8gKGW3xKN R #5i) aq``(epoch, row)`` for ``task_id``'s ``table`` rows timed at/after ``since`` (after, if ``strict``). SQLite sorts TEXT above every INTEGER, so the indexed range also returns every text stamp (written outside the mutators, #97455); those are decided on their parsed epoch. An unreadable time is dropped (never guards or lifts) unless the caller substitutes ``unreadable``.zSELECT z FROM z WHERE task_id = ? z AND r>z>=z ?N)_GUARD_TIME_COLSrrzrB) rrrrrrrrcolr*rres &&&&&$$$ r2r r s 5 !C C5fUG+>ugU3%qX^QTdhPiik lC||C5!12 MM#a& !)J =6aiqz&L 3sA(B/B7B?B Bc$V^8dQhRRRRRR/#)r4rr7r#r&r6r$r()r8s"r2r9r9s! > >C >- >D >r1cVR8wdR#\P!V/4p\V\4'dVP R4R8XdR#VP R4p\ V4;'dRV9;'d VR,V8g#)uOnly an ``assigned`` event that moves the card to a DIFFERENT profile is a handoff. A no-op re-assign (dev→dev via CLI/dashboard/``reassign --reclaim``), an unassign, or the dispatcher's own ``kanban.default_assignee`` write would otherwise lift ``active_pr`` for the very implementer that opened the PR. Events without ``from`` (written before it was recorded) are not trusted as handoffs — fail closed.assignedTsourcekanban.default_assigneeFrfrom)rz_json_orrrr=r$)rr#datatos&& r2rrsu z << $D dD ! !TXXh%7;T%T * B 8 = =$ = =4<2+==r1cV^8dQhRR/#)r4r6zOptional[Callable[[str], bool]]r()r8s"r2r9r9s;r1czaaa^RIHoHo\ S4oSfS#RTTT3RllpT# \dR#i;i)u``hermes_cli.profiles.profile_exists``, or ``None`` when it cannot be imported (local import avoids a cycle; callers fall back to trusting the assignee). When ``kanban.dispatch_profiles`` is set (#110995) the returned predicate additionally requires the assignee to be listed, fail-closed — so a card assigned to ``default`` is only claimable by homes that opted into it. Foreign assignees land in the existing ``skipped_nonspawnable`` bucket. )normalize_profile_nameprofile_existsNc V^8dQhRRRR/#)r4namer7r6r$r()r8s"r2r9(_profile_exists_fn..__annotate__sAASATAr1cv<S!V4pTS9;'d\S!T44# \dR#i;i)F)rr$)r.canon allowlistr+r,s& r2_gated"_profile_exists_fn.._gatedsE *40E !@@d>$+?&@@  s ) 88)hermes_cli.profilesr+r,r_dispatch_profile_allowlist)r3r2r+r,s @@@r2_profile_exists_fnr7sKN,,BCIAA M s + ::cV^8dQhRR/#)r4r6zOptional[frozenset]r()r8s"r2r9r9s..;N.r1c@^RIHpV!RR7;'g/PR/4p\T\4'dRT9dR#TR,pTe-\T\4'd@TP4'g*\P P R4\4#\T\\34'dTUu.uFp\T4NK upM\T4P!R 4p\#4pTFpTP%T!T44K \T4# \dIp\P P R\T4PT4\4uRp?#Rp?ii;iuupi \&dKi;i) uPer-home claim allowlist ``kanban.dispatch_profiles`` (#110995). On a shared board (one ``kanban.db`` mounted across several Hermes homes), every home's ``profile_exists`` returns True for ``default`` — the root profile every home has — so a card assigned to ``default`` is claimable by every home's dispatcher. A home opts out of foreign claims by declaring which assignees it may claim:: kanban: dispatch_profiles: ["sage", "researcher"] # or "sage,researcher" Returns ``None`` only when the key is absent from the user config (upstream behavior: any existing profile is claimable). A present value is fail-closed: an empty list, ``null`` or a bare ``dispatch_profiles:`` claims nothing. The user layer is read without the ``DEFAULT_CONFIG`` merge (whose ``None`` placeholder would make the key look present in every home), and a config read that raises also claims nothing — a corrupt config on a shared board must never widen this home's claim scope silently (#113620). )load_user_config_effectiveT) fail_closedkanbanuskanban: could not read kanban.dispatch_profiles (%s: %s) — this home claims no cards until the config is readableNdispatch_profilesukanban: kanban.dispatch_profiles is present but empty — this home claims no cards; omit the key to allow any existing profile,)hermes_cli.config_effectiver:r=rrzrrDtyper) frozensetrr r7rr/tuplersetaddr)r+r:r<excrpnnamesalloweds& r2r6r6sT( J,>DD"II(TVW fg & &*=V*K $ %C {z#s++CIIKK  J {%/dE]%C%CS !SSVS !SX[I\EeG   KK.q1 2 W /   E I   {  "    s:D3D3 F  F3 F>=F;FF FFcV^8dQhRR/#rur()r8s"r2r9r9sWWCWr1c^RIHp\T4pTfR#T'dRP \ T44#R# \dpRT R2uRp?#Rp?ii;i)aHuman-readable resolution of ``kanban.dispatch_profiles`` for this home. Surfaced by ``hermes kanban diagnostics`` so an operator on a shared board can see what a home believes it may claim (#113620): ``any`` (key absent), the sorted allowed names, or ``none (fail-closed: ...)``. )r+z)none (fail-closed: profiles unavailable: )Nr r<unone (fail-closed: kanban.dispatch_profiles is present but names no valid profile, or the config could not be read — omit the key to allow any))r5r+rr6rBr?)r+rEr2s r2"dispatch_profile_allowlist_summaryrLsaB>,,BCIyy *++ V B:3%qAABs= AA AAc$V^8dQhRRRRRR/#)r4rrrr7r6r$r()r8s"r2r9r9s' @ @+ @S @T @r1caVPRV34P4pV'gR#\4oSfR#\;QJdV3RlV4F 'gK R# R#!V3RlV44#)zdSELECT DISTINCT assignee FROM tasks WHERE status = ? AND assignee IS NOT NULL AND claim_lock IS NULLFTc3B<"TFpS!VR,4xK R#5i)rNr()rrr,s& r2r!_has_spawnable.. s?$3~c*o..$s)rrr7r )rrrr,s&& @r2_has_spawnablerQsj << K   hj  ')N 3?$?33?3?3?$? ??r1c V^8dQhRRRR/#r4rrr6r$r()r8s"r2r9r9s))0)T)r1c\VR4#)a"True iff a ready+assigned+unclaimed task maps to a real Hermes profile. Lets health telemetry tell "stuck" (``0 spawned`` with spawnable work) from "correctly idle" (only control-plane lanes waiting on ``claim_task``). Falls back to "any assigned" when ``profile_exists`` is unimportable. rrQrs&r2has_spawnable_readyrWs $ ((r1c V^8dQhRRRR/#rSr()r8s"r2r9r9s**1*d*r1c\VR4#)z2:func:`has_spawnable_ready` for the review column.rrUrVs&r2has_spawnable_reviewrZs $ ))r1cV^8dQhRR/#)r4r6r$r()r8s"r2r9r9sr1c^RIHp\V!4;'g/PR/4PRR44# \dR#i;i)zWhether review tasks dispatch automatically. Default true (Hermes ships ``sdlc-review``); operators disable it for human-only review boards.  load_configr<review_dispatchT)hermes_cli.configr^r$r=rr]s r2review_dispatch_enabledrasM1[]((b--h;??@QSWXYY sA&A AAicV^8dQhRR/#)r4r6rr()r8s"r2r9r99s  t r1cX^RIHpV!4;'g/# \d/u#i;i)uBest-effort system memory snapshot (KiB values), ``{}`` when unknown. Local import keeps ``kanban_db`` importable without the gateway package. Module-level indirection is also the test seam — conftest patches this to ``{}`` so results don't depend on the CI runner's live memory.  sample_memory)gateway.lifecycle_ledgerrerrds r2_system_memory_samplerg9s.:$$"$  s ))c V^8dQhRRRR/#)r4sampleOptional[Mapping[str, Any]]r6rr()r8s"r2r9r9Gs ] ]+F ]R_ ]r1cVf \4pVPR4p\V\4'g\V\4'dV^8:dR#VR,\ ,p\ \\V\44#)zMemory-derived default for ``kanban.max_in_progress`` when unset: ``clamp(MemTotal / MEMORY_GUARD_MB_PER_WORKER, FLOOR, CEILING)``. Returns ``None`` (no cap) when total memory is unknown, so macOS/Windows dev machines are unaffected. N mem_total_kibi) rgr=rr$rMEMORY_GUARD_MB_PER_WORKERr DERIVED_MAX_IN_PROGRESS_FLOORminDERIVED_MAX_IN_PROGRESS_CEILING)ri total_kibworkerss& r2derive_default_max_in_progressrsGsh ~&( ?+I)T""*Y*D*D UVD %??G ,c';Z.[ \\r1c V^8dQhRRRR/#)r4 configuredrr6r()r8s"r2r9r9Vs,, ,-,r1c"VeV#\4#)zEffective global concurrency cap: explicit config wins, else the memory-derived default. All config-parsing callers route through this so both paths agree. )rs)rus&r2resolve_max_in_progressrwVs  ) ++r1cV^8dQhRR/#)r4r6rr()r8s"r2r9r9`s''M'r1c^RIHpV!4;'g/PR/4PR4pTfR#\ T4pT^8dT#R# \dR#i;i \ \ 3dR#i;i)zRead ``kanban.max_in_progress`` from config, or None when unset/invalid. Shared so every dispatch entry point agrees on "explicitly configured": a positive integer wins, anything else falls through to the derived default. )load_config_readonlyr<max_in_progressN)r`rzr=rrrr)rzrpivals r2configured_max_in_progressr}`s :#%++002>BBCTU {3x194&$&  z "s'A"A A' A$#A$'A<;A<c V^8dQhRRRR/#)r4rrr6rr()r8s"r2r9r9ts0Sr1c\VPR4P4^,4# \d^#i;i)u Number of tasks in ``status='running'``. Used by the multi-board sweep to count OTHER boards' workers against the host-level budget — the memory-derived cap bounds the machine, not the board. Fails open to 0 so a broken board doesn't brick dispatch on healthy ones. z3SELECT COUNT(*) FROM tasks WHERE status = 'running')rrrrrVs&r2count_running_tasksrtsE LLE hj   s.1 AAc V^8dQhRRRR/#)r4rr&r6rr()r8s"r2r9r9s""M"S"r1c \\P!VR7P4P 44p\P !RR7p^pTFpTPR4;'g\Pp\P!TR7P4p\TP 44pTe Yq8XdKwTP4'gK\P!TR7pT\T4, p\P!\ 4;_uu_4TP4RRR4K T# \ dRpEL&i;i \ d^#i;i +'giEK2;i \P!\ 4;_uu_4TP4RRR4i +'gii;i;i \ dEKi;i)aNTotal ``running`` tasks across every board EXCEPT ``board``. Caps bound the HOST, but each board's tick only sees its own DB; without this a derived cap of N gets multiplied by the number of active boards. Boards are matched by resolved DB path, so ``HERMES_KANBAN_DB`` (pins every board to one file) yields 0. Fails open per board. rNF)include_archivedslug)r7rzkanban_db_path expanduserresolver list_boardsr= DEFAULT_BOARDexists_kbcconnectrrErFclose) r current_pathboardstotalmetarpathresolvedothers & r2 count_running_tasks_other_boardsrsr3--E:EEGOOQR %8 Exx443#4#4 %%D1<<>D4<<>*H'H,D;;==LLt,E ",U33((33KKM43" L1    433Z((33KKM4333   s#r1cVf \4pV'gR#^RIHpV!VPR4VPR44# \dR#i;i)u9Classify system memory pressure: ok/elevated/critical/unknown. Reuses :func:`gateway.memory_status.classify_pressure` so "critical" matches the dashboard banner and lifecycle-ledger OOM heuristics. ``unknown`` (non-Linux, read failure) imposes no restriction — never brick dispatch where /proc is unavailable. rm)classify_pressuremem_available_kibrl)rggateway.memory_statusrr=r)rirs& r2_memory_pressure_levelrsQ~&( ; ,?!@&**_B]^^ s,A AAspawn_fn ttl_secondsdry_run max_spawnr{default_assigneemax_in_progress_per_profilereconcile_orphansTcHV^8dQhRRRRRRRRRRR R R R R R RR RRRRRR/ #r4rrrrrr$rr{rrrPrr&rrrr6rr()r8s"r2r9r9s88 8 8  8  8#888 8$8"/888r1c aaaaaaaaaa a a RVVV VVVV VV VVV3 Rllp \P!SR7p \P !T 4;_uu_4pT'g\ RR7pMT !4p\P!ST 4RRR4\P!XSSR7T# \d&T !4p\P!TSSR7Tu#i;i +'giL^;i)aRun one dispatcher tick under the board's single-writer lock. Wraps :func:`_dispatch_once_locked` in the non-blocking :func:`_dispatch_tick_lock` so two dispatchers on one ``kanban.db`` never race a write tick on WAL frames. The loser returns an empty ``DispatchResult`` with ``skipped_locked=True`` and writes nothing; the lock is keyed on the resolved DB path so unrelated boards tick in parallel. cV^8dQhRR/#)r4r6rr()r8s"r2r9#dispatch_once..__annotate__s  . r1c2< \SS S SSSSS SSSSR7 #)) rrrrr{rrPrrrr)_dispatch_once_locked) rrrrrr{rrrrrPrs r2 _locked_tick#dispatch_once.._locked_ticks4$ #+'"7-(C/  r1r)rrT)r%N)rzrr_fire_dispatch_tick_hookr_dispatch_tick_lockr_maybe_checkpoint_wal)rrrrrr{rrPrrrrrdb_pathresultheldsfddddddddddd r2 dispatch_oncers.   $$51  ! !' * *d#48F!^F  & &tW 5 +  ugF M  $$V5'J  + *sB.4C!.-CC! C1 c(V^8dQhRRRRRRRR/# r4rr workspacer7rr&r6rr()r8s"r2r9r9s) ) )4 )C ) )R_ )r1c^RIpVPV4pRVP9d V!WVR7#V!W4# \\3d T!Y4u#i;i)zBack-compat: older spawn_fn signatures (and test stubs) accept only ``(task, workspace)``; pass ``board`` only when the callable supports it.Nrr)inspect signature parametersrr)rrrrrsigs&&&& r2_call_spawn_fnrs_)) cnn $D59 9(( z ")(()s+::AAcHV^8dQhRRRRRRRRR RR R R R RRRRRR RRRR / #)r4rrrz sqlite3.Rowrr7rrrrr$rrrr&rrper_profile_capper_profile_runningzdict[str, int]r6r()r8s"r2r9r9s|| | ||  |  ||| ||#|(| |r1c  a a VR,p \4p V eV !V4'gVPPV 4V'g\P!V4;_uu_4VP RV 34P 4pVe3VR,R8wg%VR,\P!RV/48wd\P!W RRV/4RRR4R#R#S e9S PV^4pVS 8d VPPWV34R#\W VR 7pVekVPPV V34V'gD\P!V4;_uu_4\P!W R R V/4RRR4R#R#R V V 3R llpV'd(VPPWR34V!V4R#VR8Xd\PM\PpV!W VR7pVfR#RpVP R8Xd\"P$!VVR7wppM\"P&!VVR7p\"P0!VVP,\3V44VP R8Xdf\"P4!TVP,T;'g<VP6;'gRP94;'gRVP, 24\"P:!VVP,VP 4VR8Xd<\=\>PA.VPB;'g.ORN44Vn!\EV eT M\FV\3V4V4pV'd8\IVVP,\KV4VPLVPNR7\PP!VV\3V4VVR7VPPVP,VPR;'gR\3V434V!VPR4R# +'giR#;i +'giR#;i \(dVp\+TTP,RT 2RTRRR7'd&TP.PTP,4Rp?R#Rp?ii;i \(dp^RI*H+p\YTT4pT'd,\PZP]RTP,T4\+TTP,\3T4RTRRTR7'd&TP.PTP,4Rp?R#Rp?ii;i)zGuard, claim, resolve the workspace and spawn one ready/review row. Returns True when a spawn slot was consumed (real or ``dry_run``); every skip is recorded on ``result``. rNzaSELECT kind, payload FROM task_events WHERE task_id = ? ORDER BY created_at DESC, id DESC LIMIT 1rrr#rFrr"rGc V^8dQhRRRR/#)r4r.r7r6rUr()r8s"r2r9)_dispatch_lane_task..__annotate__DsMM3M4Mr1c`<Se)V'dSPV^4^,SV&R#R#R#r)r=)r.rrs&r2 _count_spawn)_dispatch_lane_task.._count_spawnDs2  &4(;(?(?a(H1(L  %,0 &r1rTr)rworktreerz workspace: r)r;rr>r?zwt/z sdlc-review)rr)RestartSafeScopeUnavailablezIkanban dispatcher: spawn of %s deferred, host cannot place the worker: %s)r;rr>r?r)/r7rrArzrrr _json_or_nullrr=rrr"rclaim_review_task claim_taskworkspace_kind_kbw_resolve_worktree_workspaceresolve_workspacerrIrrset_workspace_pathr7set_branch_name branch_namer_maybe_emit_scratch_tipr/rfromkeysskillsr_default_spawnrrrr_fire_worker_spawned_hookrtools.process_registryrrrrD)rrrrrrrrrrrrrr,lastr guard_reasonrclaimclaimedresolved_branch_namerrErSrrs&&&&$$$$$$dd r2_dispatch_lane_taskrs&$iG ()N!.*B*B##**73 t$$||@BIMMUXZLDL4J$J ?c.?.?X@V.WW%%d5KjZbMcd %"%))(A6 o %  - - 4 4g5Q R&t4@L%%w &=>t$$!!$1BX|D\]%MM w"56X%)X%5C ! !3>>ED{;G #  ! !Z /.2.N.Nw^c.d +I+..weDI D'**c)n=+ T7::/C0A0AH[H[HaHa_aGhGhGj0A0Apst{t~t~soA B  wzz73I3IJ xdmm,Tw~~/C/C,Tm,TUV)=X>SZ\_`i\jlqr  D'**c#hw?U?Ubibtbt u %%dGS^SPUV wzz7+;+;+A+Ar3y>RSW%%&Y%$&%$,   '** C51"-t]a      & &wzz 2 6 F$C)DE  HH  hjqjtjtvy z  '**c#h"-t]a)     & &wzz 2sX)A2Q7Q.Q,>Q,'B6S$S Q  Q) , S 7A SS  U;BU66U;c ,V^8dQhRRRRRRRRRR/#) r4rrrr7rrr$r6r()r8s"r2r9r9s4 '*69GK r1c XV'dR#\P!V4;_uu_4VPRW!34\P!WRRVRR/4RRR4R# +'giR#;i \d'\P P RY!RR 7R #i;i) a Persist ``kanban.default_assignee`` on an unassigned ready row. Mutating the row keeps board state honest: the task is legitimately owned by the default, not "unassigned but secretly routed". ``dry_run`` reports without writing. Returns False when the write failed. TzRUPDATE tasks SET assignee = ? WHERE id = ? AND (assignee IS NULL OR assignee = '')r"rr#r$Nz?kanban dispatch: failed to apply default_assignee=%r to task %srF)rzrrrrrr)rrrrs&&&$r2_apply_default_assigneers ]]4 LL:#    zXx1JK  ! !!    M    s.A80A$A8$ A5 /A85A88-B)(B)c4V^8dQhRRRRRRRRRR R R R R /#)r4rrrrrPrrrr$rr&r6rUr()r8s"r2r9r9sWMM M M M  M  M M Mr1c\4\V4Vn\P!WR7VnV'd\ V4Vn\WR7Vn \WR7Vn VPP\\R.44VPP\\R.44\!V4Vn\P$!WR7VnR#)zEReclaim stale/orphaned/crashed/timed-out running tasks, then promote.)r)rPrrrN)rrrrzrelease_stale_claimsrrcrr[r!rrrextendrr#rOr recompute_readyr)rrrPrrrs&&$$$$r2_run_reclaim_phasers%:4%@F"//RF$>t$D!'ZFL+D>FN w'=?SUWXY w'=?SUWXY*40F))$LFOr1c 0V^8dQhRRRRRRRRRR R R /#) r4rrrrrrr{rr&r6ztuple[bool, Optional[int]]r()r8s"r2r9r9sD:: : : : # :  : :r1c^pRpVfVe \V4pVeWR8dR#W%, pVe/V\V4,pWs8dR#W7, pVeWh8dTp\4p V R8Xd(Wn\P P R4R#V R8Xd3WnVeV^8d"\P P R4^pRV3#)u``(may_spawn, spawn_budget)`` for this tick; ``budget None`` = uncapped. ``max_spawn`` is a live per-board concurrency cap (running + this tick's spawns), not a per-tick budget — a per-tick reading would grow concurrency by N every tick. ``max_in_progress`` is a HOST-level cap: running workers on every other board count against the same budget, else N boards multiply the cap by N — exactly the fan-out the memory-derived default exists to prevent. Ncriticalznkanban dispatch: system memory pressure is critical; spawning no new workers this tick (deferred, not dropped)elevatedz_kanban dispatch: system memory pressure is elevated; limiting to at most 1 new worker this tickT)FN)rrrr'rzrrD) rrrr{r running_count spawn_budget total_running remainingrDs &&$$$ r2_tick_spawn_budgetrs&M"&L ;+D1   %  0 "%(H(OO  + #3  <#;$L &'H:!)  H :!)   F r1c V^8dQhRRRR/#)r4rr&r6r()r8s"r2r9r9" s   - r1cT;'gRP4;'gRpV'd\4pVeV!V4'gR#V#)al``kanban.default_assignee`` when it names a real profile this home may claim (``kanban.dispatch_profiles`` gated, same predicate as the spawn gate). Otherwise ``None`` so an unassigned shared-board card is never written to. When the profiles module isn't importable trust the operator's config: the downstream check still buckets a missing profile as nonspawnable.rN)rr7)rr.r,s& r2_resolve_default_assigneer" sE  " " ) ) + 3 3tD +-  %nT.B.B Kr1cHV^8dQhRRRRRRRRRRR R R R R R RR RRRRRR/ #rr()r8s"r2r9r94 see e e  e  e#eee e$e"/eeer1c h\4p \W VWkVR7\W WEVR7wrV 'gV #\VR4p\ 4'd \VR4M.p\ V \ 4'd V ^8dT MRp/pVe5VPR4Fp\ VR,4VVR,&K! TpVe0V^8d)\VVVVR 7'd\V^, ^4p\W2VWaVVR 7p\V 4p ^pVFpVe VV8dMVR,pV'glV 'd\VVR ,WR 7'g%V PPVR ,4KhT pV PPVR ,4\!VVVV 3R R/VB'gKV^, pK VFmpVe VV8dV #VR,'g%V PPVR ,4KD\!VVVR,V 3R R/VB'gKdV^, pKo V #)a5One dispatcher tick: reclaim stale/crashed running tasks, promote todo -> ready, then atomically claim each spawnable ready/review row and call ``spawn_fn(task, workspace_path, board) -> Optional[int]``, recording the PID so later ticks catch crashes before the TTL. Cap semantics: :func:`_tick_spawn_budget`.)rPrrr)rr{rrrNzmSELECT assignee, COUNT(*) AS n FROM tasks WHERE status = 'running' AND assignee IS NOT NULL GROUP BY assigneerFr)rr)rrrrrrrr)rr)rrrrrarrrrr rrrrrArr)rrrrrr{rrPrrrrr may_spawnr ready_rowsrrrprow ready_budget lane_kwargsrr row_assignees&$$$$$$$$$$$ r2rr4 sS( F ,A#PU1 RWI  D'*J1H0I0I*T8,rK .44 '! + 2 +-"LL  D 58S N Z 0 1   LL1$49N k'=P::Return an absolute filesystem path for a resolved Hermes shim.)rrrisabsabspath)rexpandeds& r2_absolute_hermes_pathr s@ww!!$'Hww}}X..8MBGGOOH4MMr1c V^8dQhRRRR/#)r4rr7r6r$r()r8s"r2r9r9 s  C D r1cz\PPV4pVPR4;'g\PP V4;'g^\ \PP V44;'g.RV9;'g!\ \P!RV44#)zDReturn true when a command override is an explicit path, not a name.~\z ^[A-Za-z]:) rrrrrr$dirnamerhmatch)rrs& r2_looks_like_pathr sww!!%(HC  3 3 77== " 3 3 ) * 3 3 8  3 3 1 2 r1c V^8dQhRRRR/#)r4rr7r6r$r()r8s"r2r9r9 s3333r1c@VP4PR4#)zEReturn true for Windows shell/batch shims that should not be argv[0].)z.cmdz.bat)rjendswithrs&r2_is_windows_batch_shimr! s ::< !1 22r1c V^8dQhRRRR/#)r4commandr7r6rr()r8s"r2r9r9 s<<< >&"-33BJJ?'S. GG&&w/ &w/D Y5Iww~~i((cooo9VXV]V]A^A^  0 @ r1c V^8dQhRRRR/#)r4rr7r6rr()r8s"r2r9r9 s))C)I)r1c|\P'd\V4'd \4#\ V4.#)zargv for a resolved Hermes executable path. Windows batch shims (``.cmd``/``.bat``) are unsafe as argv[0] because the argument vector includes task-derived values; prefer the module form.)rzrr!rrr s&r2_hermes_path_argvr9 s/ 1$77"$$ !$ ' ((r1cV^8dQhRR/#r r()r8s"r2r9r9 s!!!!i!!r1c^RIp^RIp\PP RR4P 4pV'dE\ V4'd \V4#\V4pV'd \V4#\4#VPPR4e \4#\P'd \R4MVPR4pV'd \V4#\4# \dL\i;i)aResolve the ``hermes`` invocation as argv for ``Popen``: ``$HERMES_BIN`` (path-like -> absolute; bare names keep PATH semantics, never a same-directory file), then the running interpreter's ``sys.executable -m hermes_cli.main`` (exactly this install; also covers shim-less cron, systemd ``User=``, launchd), then ``which("hermes")`` (Windows: safe PATH search, batch shims fall back to the module form) only when ``hermes_cli`` is not importable. The module argv must win over PATH: a PATH-first lookup lets an attacker-planted ``hermes`` shadow the running install (#111569). Mirrors ``gateway.run._resolve_hermes_bin``; local because ``hermes_cli`` sits below ``gateway`` in the dependency order. N HERMES_BINr hermes_clihermes)importlib.utilshutilrr(r=rrr9r6rutil find_specrrzrwhich) importlibr@env_binresolved_env_bin hermes_bins r2_resolve_hermes_argvrH sjjnn\2.446G G $ $$W- --g6 $%56 6"$$ >> # #L 1 =&( ( > 25#H-fllS[F\J ,,      s&C;; D D c$V^8dQhRRRRRR/#)r4r3rcurrent_timeoutr&r6r()r8s"r2r9r9 s&&"r1cHVfR#\V4pT^8:dR#\^T\, 4pT'd#\\ T4P 44M^pYC8dR#\ T4# \\3dR#i;i \\3d^pL@i;i)zReturn a worker-scoped TERMINAL_TIMEOUT override, if needed. When ``max_runtime_seconds`` exceeds the terminal tool's default timeout, raise only the child's default so a long command isn't killed by the generic terminal default first. N)rrrr %KANBAN_TERMINAL_TIMEOUT_GRACE_SECONDSr7r)r3rJruntimedesiredexistings&& r2_worker_terminal_timeout_envrP s")*!|!WDDEG8G3s?+1134Q w< z " z "s" A3,B 3BB B! B! bind_homec V^8dQhRRRR/#)r4 hermes_homer7rQr$r()r8s"r2r9r9& s*3*3s*3$*3r1c#"^RIHpHpHp^RIHpHpHp^RIH pH p ^RI H p H p \V4p \V P!44\\V!44P!448Hp R;p;ppV'dV!\V 44MRpT!V 'd V !V 4MV!V 4V 'dRM \V 4R7pV'dT!Y'dV !4MRR7MRpRxVe V !V4Ve V!V4Ve V!V4R#R# Te T !T4Te T!T4Te T!T4ii;i5i)uBind an assigned profile's runtime scope (secrets + terminal policy, optionally home) for one dispatch-side read or spawn-env build. The dispatcher runs detached from any turn, so nothing binds a profile for it: ``load_config``, the toolset probes' ``get_secret`` reads and ``build_subprocess_env``'s passthrough resolution all fall back to the LAUNCH profile's ambient ``os.environ`` / ``TERMINAL_*``. Binding was previously conditional on ``is_multiplex_active()``, so on a single-profile host a worker for profile B was built entirely from the dispatcher's own environment. ``bind_home=False`` for the spawn-env build: which variables may cross into a child is the DISPATCHER's ``terminal.env_passthrough`` policy (#109494, read through the home override) — only their VALUES come from the assignee's scope, so that branch binds the secret scope alone. Toolset resolution binds the home and the terminal policy, as it always has. The secret mapping is never widened: a profile that is not this process's own home gets its own ``.env`` + external sources ONLY, while the launch home keeps its established env-over-``.env`` precedence (``launch_secret_scope``) so systemd / ``op run`` injection still resolves for a standalone dispatcher. )build_profile_secret_scopereset_secret_scopeset_secret_scope)get_process_hermes_homereset_hermes_home_overrideset_hermes_home_override)install_profile_terminal_scopereset_terminal_scope)launch_secret_scopelaunch_terminal_envN) profile_home) env_overlay)agent.secret_scoperUrVrWhermes_constantsrXrYrZtools.terminal_scoper[r\!tui_gateway.launch_profile_policyr]r^rr7r)rSrQrUrVrWrXrYrZr[r\r]r^homeis_launch_home home_token secret_tokenterminal_tokens&$ r2_worker_profile_scoperj% s,*dcnnYZ  D(C5L5N0O0W0W0Y,ZZN155J53H040 /   S    s4A,AA,A, A) $A,, B#7!BB#zset[str]_retagged_workspace_rootsc V^8dQhRRRR/#)r4workspaces_root_pathr7r6rUr()r8s"r2r9r9q sPPPPr1c*V\9dR#^RIHpHpV!4pVP V4V!V4\P V4R# T!T4i;i \ d,p\PPRT4Rp?R#Rp?ii;i)aReclaim pre-tag worker rows in state.db so they leave the session lists. Best-effort: the durable gate is ``state_meta`` in ``retag_kanban_worker_sessions``; the in-process set avoids reopening state.db on every spawn. A tick must never fail because a session DB was busy or missing. N)acquirerelease_or_closez0kanban worker: legacy session retag skipped (%s)) rshermes_state_registryrwrxretag_kanban_worker_sessionsrDrrzrr)rurwrxdbrEs& r2_retag_legacy_worker_sessionsr|q s88 PCY !  + +,@ A R !%%&:; R P I3OOPs-AAA AA B' B  Bc(V^8dQhRRRRRRRR/#) r4rr profile_argr7rSr&r6rr()r8s"r2r9r9 s.EEtE#EMEiEr1c(^RIpVPRR.#)zGThe profile owner executes; this subprocess only waits for its receipt.Nr zhermes_cli.kanban_worker_clientr )rr~rSrs&&& r2 _worker_argvr s NND"C DDr1c V^8dQhRRRR/#)r4rrrr&r()r8s"r2r9r9 s   4   r1c\P!VR7pVPRRR7W P R2, p^RIHpHpV!V.V!4O5!\VR4#)zAppend-mode per-task log (a re-run on unblock appends, never overwrites), rotated first. Anchored at the board root (not the shared kanban root) so `hermes kanban log` reads its own file and boards sharing task ids don't collide.rT)parentsexist_okz.log)rotate_worker_logworker_log_rotation_configab)rzworker_logs_dirmkdirrhermes_cli.kanban_worker_logrrr)rrlog_dirlog_pathrrs&& r2_open_worker_logr sX !!.G MM$M.GG9D))HZh>!;!=> $ r1c$V^8dQhRRRRRR/#)r4rrr#rr6r()r8s"r2r9r9 s!# # D# 9# # r1c^RIHpVPf9V!VRVP R2RRR7pVPR8wd \ R4hV#V!VRVP RVP 2RRR7P #) a:Wrap a systemd-hosted dispatcher's worker in the shared restart-safe scope. Kanban workers are long-lived agentic runs that outlive the dispatcher tick, so they never take cron's degraded mode under the managed gateway: ``require_restart_safe_scope=True`` makes the helper raise ``RestartSafeScopeUnavailable`` there (an infrastructure spawn failure the dispatcher does not charge to the card). Under any other systemd unit (``Type=oneshot`` dispatch timers, #113612) ``outlives_parent=True`` gets the worker its own scope so the unit's cgroup teardown cannot kill it. )restart_safe_gateway_child_argvzkanban-z -run-missingT) unit_suffixrequire_restart_safe_scopeoutlives_parent in_processzbcannot create restart-safe systemd scope for Kanban worker: the claimed task has no current run idz-run-)rrrrmode RuntimeErrorargv)rr#rdispatchs&& r2_restart_safe_worker_argvr sG "3 !$'',7'+   ==L (9  *dggYeD,?,?+@A#'    d  r1c(V^8dQhRRRRRRRR/#rr()r8s"r2r9r9 s/PPP#PPR_Pr1c ` VP'g\RVP R24h^RIHpHpV!VP4p^RIHp^RIH pH pH p V!V4p \V 4;'d V!V 4p V 'd\V RR7M\P !4;_uu_4T!V!4;'gT R R 7p RRR4^R IHp V FpX P'VR4K V 'dV X R &V !W4VP('dVP(X R &VPX R&WR&RV R&V'dO\*P,P/V4'd*\*P,P1V4'dWR&VP2'dVP2V R&VP4e\7VP44V R&VP8'dVP8V R&VP:'d5RV R&VP<e"\7\?VP<44V R&R$F2p\AVPBV PEV44pVfK.VW&K4 \7\FPH!VR74V R&\7\FPJ!VR74V R&\FPL!V4;'g\FPN!4T R&W\R&^RI(H)pV P'VR4V P'RR4\UWV PER 44p^R I+H,pV!V \[\\4P_4P`P`4p \cVV4p^R!I2H3pV!V 4p \iW4p\jPl!T\*P,P1V4'dTMR\jPnT\jPpT R \FPr'd\jPtM^R"7p\FPr'dT\zTP|&TP|# \dRp ELi;i +'giEL;i \dTPw4\yR#4hi;i)%aFire-and-forget ``hermes -p chat -q ...`` subprocess. Returns the child's PID so the dispatcher can detect crashes before the claim TTL expires; completion is still observed via the worker's own ``complete`` / ``block`` transitions. ``board`` pins the child's ``HERMES_KANBAN_DB`` / ``HERMES_KANBAN_BOARD`` / workspaces_root to the board the task was claimed from, so workers cannot see other boards. ztask z has no assignee)r+resolve_profile_env)is_multiplex_active)_is_routed_homebuild_subprocess_envstrip_launch_profile_envNF)rQT) scrub_secretsinherit_profile_home)_VAR_MAP HERMES_HOME HERMES_TENANTHERMES_KANBAN_TASKHERMES_KANBAN_WORKSPACEr<HERMES_SESSION_SOURCE TERMINAL_CWDHERMES_KANBAN_BRANCHHERMES_KANBAN_RUN_IDHERMES_KANBAN_CLAIM_LOCK1HERMES_KANBAN_GOAL_MODEHERMES_KANBAN_GOAL_MAX_TURNSrHERMES_KANBAN_DBHERMES_KANBAN_WORKSPACES_ROOTHERMES_KANBAN_BOARDHERMES_PROFILE)DELEGATED_CHILD_ENV_MARKER HERMES_TUI)pin_hermes_tree_on_pythonpath)systemd_user_bus_env)cwdstdinrrenvstart_new_session creationflagszv`hermes` executable not found on PATH. Install Hermes Agent or activate its venv before running the kanban dispatcher.)TERMINAL_TIMEOUTTERMINAL_MAX_FOREGROUND_TIMEOUT)?rrrr5r+rrartools.environments.localrrrrr$rjrE nullcontextgateway.session_contextrr`tenantrrrisdirrrr7r goal_modegoal_max_turnsrrPr3r=rzrworkspaces_root_normalize_board_slugragent.delegation_contextrrcron.scheduler_worker_envrr__file__rparentrrrrrPopenrSTDOUTrCREATE_NO_WINDOWrrrMrS)rrrr+rr~rrrrr_routedrrr\varoverridercmdrrlog_frs&&$ r2rr s ===5 )9:;;O(7K6hh*;7 ,  A AOL$AFCO   >%%' ( ("-/996!%  (1 T)M !3 {{{#{{O $C%.!"$,C RWW]]9--"''-- 2J2J'N &*&6&6 "# &&)$*=*=&> "# *.// &' ~~~), %&    *25c$:M:M6N2OC. /F/0H0H#''RU,W  CHG "#"4"45"ABC+.s/B/B/O+PC'("%!:!:5!A!\!\SEZEZE\C(CGG &-GGL$ t#''-*@ AC H 'T(^-C-C-E-L-L-S-S TC $D# .C; s #C T )E  WW]]955 4$$$$"9<*55a  $ '+488$ 88Ow   ( ( (H    ^   s1Q,Q2"A/RR Q/.Q/2 R 'R-intervalgN@ stop_eventon_tickc(V^8dQhRRRRRRRR/#) r4rfloatrrrrr6rUr()r8s"r2r9r9Z s24*4*4*4* 4* 4*r1c a^RIpSfVP4oV3RlpVP4VP4JdgRF`p\ \ VR4pVfK\ P!\\4;_uu_4\ P !W4RRR4Kb SP4'g\\44p \ P!\P!44;_uu_4p \!V VV VR7p RRR4Ve5\ P!\"4;_uu_4V!X 4RRR4MSP)VR7KR# +'giEK8;i +'giLt;i +'giLN;i \"d^RIp T P'4Lri;i)uRun the dispatcher in a loop until interrupted. Calls :func:`dispatch_once` every ``interval`` seconds; exits cleanly on SIGINT / SIGTERM so it is systemd-friendly. ``stop_event`` and ``on_tick`` are test hooks. Each tick resolves ``kanban.max_in_progress`` exactly like the gateway dispatcher and ``hermes kanban dispatch`` — the standalone daemon must not be the one uncapped entry point. Nc(<SP4R#r)rC)_signum_framers&&r2_handlerun_daemon.._handleo s r1)rr{r)r)SIGINTr) threadingEventcurrent_thread main_threadrrrErFrris_setrwr}closingrrrr traceback print_excwait) rrrrrrrsig_namerr{rrErs $$$d$ r2 run_daemonrZ sV __&  !Y%:%:%<<-H&(D1C((W==MM#/>=. !! "66P6RSO##DLLN33t#'$3"/ 4"((33CL43# )'">==4343 "     ! "sUE*?AF%E?.F% F F%* E< ? F F% F" F%"F%%GG) kanban_db)kanban_db_connect)kanban_db_workspacer)r)__conditional_annotations__r- __future__rrErrhrsqlite3rrr] dataclassesrrpathlibrtypingrrrr r r r hermes_cli.quiet_single_queryr hermes_cli.kanban_dbrrrLrcompile IGNORECASErr #DEFAULT_RATE_LIMIT_COOLDOWN_SECONDSr rrrKr_r^rLr.rMrQrirrrnescape MULTILINErrrrrrrrrrrrrrrr$r,rOrWr[rcrlrrprsrrrzr}rrrrrrrrrIrrrrrr rr7r6rLrQrWrZrarmrnrprgrsrwr}rrrrrrrrrrrrrrrrrr!r*r6r9rHrPcontextmanagerrjrrrCrsr|rrrrrr=rrzrrrr)rs@r2rs #  ! D)  )+% &)"jj" MM!% '*#!::4MM  >L>L >LBJ#&5727 35/4) 1"."::299/ 00<? 1  8.bW( @)* &!!""#  ],'(""J&88"& 8  8 $ 8&*8/8"#8 8'+8268#8v )|~>M M2:z&*  59 @ $ee"& e  e $ e&*e/e"#e e'+e26e#eP4a45 N 3 <$)!!H: *3*3*3Z8'*e8+P2E  # LPPn4*4* $4*/ 4*  4*  4*r(02r1