߲jUdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlmZddlmZmZmZ ddlZddlmZddlmZmZmZmZmZm Z m!Z!e jDjGde$ee%jLjLddl'm(Z(m)Z)ddl*m+Z+dd l,m-Z-dd l.m/Z/m0Z0dd l1m2Z2dd l3m4Z5dd l6m7Z7ddl8m9Z9m:Z:ddl;mZ>ej~e@ZAd-dZBdZCde$fdZDdeEde$fdZFdeGe$e$fdzfdZHde$dzfdZIGddeJZKe jdZMde$dzdeGe$e$fdzfdZNdZOiaPeEe$eQfeRd<deGe$e$fdzfd ZSd!e$ddfd"ZTdeEd#e$dzde$fd$ZUd%ZVdeWfd&ZXd'eEdeYfd(ZZdd)deEd#e$d*eedeGeYee$ffd+Z[deEddfd,Z\d-ee$ddfd.Z]Gd/d0e^Z_d1eEde`e$fd2Zad3e`e$d1eEde`e$fd4ZbdeEd1eEde`e$fd5ZcdeEd1eEd6e$deEdzfd7Zddd8lemfZfmgZgmhZhmiZimjZjmkZkmlZlmmZmmnZnmoZompZpmqZqmrZrmsZsdd9ltmuZumvZvmwZwmxZxmyZymzZzm{Z{m|Z|d:Z}d;Z~dZiZee$ejjfeRd?<iZee$eefeRd@<d.dAe$dBee!ee$fdeGfdCZiZee$efeRdD<dBedefdEZdFe$defdGZeZeeRdH<iZeEeGeEeQeGee$efffeRdI<eZeeRdJ<iZeEeGefeRdK<e jZiZeEeRdL<iZeEeRdM<iZeEeRdN<eQZdaeeRdO<gZe`eRdP<dQZdRZeZeeRdS<GdTdUe ZGdVdWZd/dXZde`eEfdYZd/dZZd.dAe$dBee!ee$fdeYfd[ZdAe$deYfd\Zd.dAe$dBee!ee$fddfd]ZdeWfd^ZiZeEeRd_<d`e$deeWfdaZdeEdeeWfdbZdeEfdcZdAe$dde$deeWdfeWddf dgZdeEfdhZdieEdjeWdeYfdkZdeEdAe$dleWdmeWd!e$ddf dnZd.doee`de`fdpZddqd!e$dreede`fdsZd.dAe$dteeQdeYfduZd.dAe$dteeQdeYfdvZdwegeWfdxeWdyeWdze j^d{egeYfdeYf d|ZdeWfd}Zd~eedejjfdZd0dZd0dZejjedefdZde$fdZdeEddfdZd.deedeYfdZdZedzeRd<defdZdeGeeffdZdedeYfdZd > R !!!s !AA c|r|sy|xsdj}|sy |j|||S#t$r7t|dd}|||}|r||k(r|j|||cYSwxYw)aPersist a non-blank, unique title for a finished cron session; returns it (None if unset). Runs BEFORE end_session()/close() so no write races the close. Duplicate title (unique-index ValueError) -> get_next_title_in_lineage(); if unavailable, raise rather than end up untitled. Centralizes the title write so the cron finally block can guarantee a non-blank, unique title is persisted before end_session()/close() tear the connection down (issues #50535, #50536, #50537): - #50535: never leaves the session blank. base_title already carries a cron-id fallback for nameless jobs; this also guards a failed write. Recover by appending a #N suffix via get_next_title_in_lineage() when supported, instead of swallowing the error and ending up untitled. - #50536: this runs synchronously in the cron finally block ahead of the session close, so no in-flight title write can race the close. Nget_next_title_in_lineage)stripset_session_title ValueErrorgetattr) session_db session_id base_titletitle next_title_fndedupeds r(_set_cron_session_titler7Es Z  2 $ $ &E  $$Z7   ,GN  &'U* $$Z9 s2=A21A2c^ txsi}t|}|ry y#t$rYywxYw)uBackup-provider clause for a provider-failure notice: "the backups failed too" vs "none configured" (most installs). Fails open to the former if config can't be read — never crash delivery. z$No backup provider succeeded either.uNo backup provider is configured — add one with `hermes fallback add`, or set a cron-wide default via `cron.model` + `cron.model_provider` in config.yaml.)rrr#)cfgchains r(_fallback_chain_phraser;fsD 6m!r"3' 5 ^ 656s  ,,jobc|jdxsijd}|dvry txsi}tt|tr|jdxsinijdd}|dkryt|jd xsdd z}||kry|jd xs|jd xsd }d|d|dS#t $rd}YkwxYw)zReview nudge when a recurring job keeps failing, else "". The failure message is delivered BEFORE mark_job_run records this run, hence stored ``failure_streak`` + 1. Threshold: ``cron.failure_nudge_threshold`` (default 3, 0 disables).schedulekind>cronintervalr+r@failure_nudge_thresholdrfailure_streaknameidzthis jobz This job has failed u^ runs in a row — worth a review. Fix its prompt/config, or pause it with `hermes cron pause z3` (resume/remove also available) to stop the noise.)getrint isinstancedictr#)r< schedule_kindr9 thresholdstreakjob_refs r(_failure_streak_nudgerPxsWWZ(.B33F;M00m!r(23(=cggfo#2 J J)1  A~ )*/a 01 4F ggfo<<*G )FFMYO< <  sAC C%$C%c< ddlm}|S#t$rYywxYw)zBoot-vs-disk revision skew for THIS process, or None. Test seam over ``gateway.code_skew.detect_code_skew``; a broken import must never take delivery down.rdetect_code_skewN)gateway.code_skewrSr#rRs r(_detect_gateway_code_skewrUs%6!!   c< ddlm}|S#t$rYywxYw)zRFull revision currently on disk; kept separate from display-shortened skew labels.rcurrent_code_shaN)rTrYr#rXs r(_current_gateway_code_sharZs%6!! rVc0eZdZdZdededdffd ZxZS)CronTickYieldeduA stale-code ticker yielded this tick to a fresh gateway. Raised by ``tick()`` BEFORE the tick lock when boot fingerprint ≠ disk, this process does NOT own the runtime lock and its live holder reports the disk revision — the stale process must stay out of the dispatch race (contention would starve the fresh ticker). Skew ``None`` never yields (fail open). Raised, not returned, so ``record_ticker_error`` sees it and ``hermes cron status`` isn't green. boot_revdisk_revrNcN||_||_t| d|d|dy)NzDCron tick yielded to a fresh gateway process (stale code: booted on , disk is at ))r]r^super__init__)selfr]r^ __class__s r(rczCronTickYielded.__init__s5     ! -z < )__name__ __module__ __qualname____doc__strrc __classcell__res@r(r\r\s'     rfr\z/stale code: booted on (\S+), disk is at (\S+)\)recorded_errorc|r|jtjsytj |}|r"|j d|j dfSdS)aO``(boot_rev, disk_rev)`` when a persisted ``ticker_last_error`` is a ``CronTickYielded``. ``hermes cron status`` runs in another process and only sees the marker text; a yielding ticker still refreshes its heartbeat, so this is the one signal that separates "stale code, firing nothing" from a healthy loop (#117275). NrE) startswithr\rg_STALE_YIELD_REsearchgroup)rnmatchs r(stale_code_yield_labelsrvsO !:!:?;S;S!T  " "> 2E/4EKKNEKKN +>$>rfg @_last_yield_logct}|yt}|y ddlm}m}t }||ry||}||j d|k7ry |S#t$rYywxYw)ut``(boot_rev, disk_rev)`` when THIS profile's tick must yield to a fresher gateway, else None. Yields only when ALL hold: code skew, this process is not the cron owner for the profile being ticked, and another live host gateway whose served set covers that profile reports a fresh heartbeat on the disk revision. Every probe failure returns None — yielding is a certainty claim, never a guess. The question is asked PER PROFILE. One process multiplexes every profile, so "do I own the runtime lock" (process-global, stamped at boot for the launch home) cannot answer it: a stale multiplexer that holds the lock must still keep ticking the profiles nobody else ticks, and must still yield the ones a fresher gateway serves. ``gateway_state.json`` is per-HOME and last-writer-wins, not per-process: during a ``--replace`` takeover both the stale and the fresh gateway stamp it, so which one this predicate reads is write-order dependent. The pid equality in ``live_gateway_ticking`` is what binds the record to the current lock holder; the takeover window itself fails open (no yield). Nr)live_gateway_tickingowns_cron_tick_forcode_sha)rUrZcron.scheduler_ownershipryrz_get_hermes_homerHr#)skewdisk_sharyrzhome holder_statuss r(#_should_yield_tick_to_fresh_gatewayrs$ % &D |(*H U! d #,T2  M$5$5j$AX$M%N K sAA A%$A%reasonctj}tjd}tjdd}||k7s|t |z t k\rt jd|||day)z?Log the yield at error level once per episode (skew signature).ratuCron tick yielded: this process is running stale code (%s) and a fresher gateway owns the runtime lock — jobs will fire from that process. Restart this one to reclaim its ticks.)rrN)time monotonicrwrHfloat_YIELD_LOG_INTERVAL_SECONDSloggererror)rr last_reasonlast_ats r(_log_tick_yield_oncersm .. C!%%h/K!!$,GfuW~!5:U U  >    "(s3Orfrc xddlm}m}m}m}m}|j dxs|j dxsd}|j dxs|}|xsdj} | j} | jdr |||Stjd| r |||S|j d s.||||| t|j d  } | | Stjd d| d d} tjdd| jjd} t| dkDr| d djdz} |||| } |j d s4tjd| r t!}||\}}| d|d|dz } | S#t"$rd }Y"wxYw)agOne-line failure notice for chat delivery (full details stay in the run output). Deterministic scheduler/script shapes are matched first (their text can contain "timed out" and would otherwise be blamed on the model service); everything else goes through the shared ``classify_api_error`` verdict and the copy table in ``scheduler_failure_copy``.r)classify_cron_failure_reasongeneric_failure_noticeinactivity_noticeprovider_failure_noticescript_timeout_noticerFrGcron jobz unknown errorzscript timed outzidle for \d+s\s*\(limit \d+s\)no_agentprovider)backup_provider_phraserNz8^(RuntimeError|Exception|ValueError|HTTPStatusError):\s*r+iz\s+ .z...z2cannot import name|modulenotfounderror|importerrorz< Likely cause: the gateway is running stale code (booted on r`u-) — run `hermes gateway restart` to fix it.)cron.scheduler_failure_copyrrrrrrHr-lowerrqrersr;subrstriplenrUr#)r<rrrrrrjob_namejob_idtextrnoticecleanedmessager~r]r^s r($_summarize_cron_failure_for_deliveryrs 88wwv=#''$-=:H WWT] &hF  $_ + + -D JJLE  *+$Xv66  yy2E: 622 77: ( f:4@#9#;cggjFY[  MffPRTVZ[`\`VabGffVS'*00299#>G 7|c$3-&&(50$Xvw?G 77: 299=u$ ,.D  !% Hh Z}XJ766 G N D s F++ F98F9g@c2ddlm}|dttS)z``cron.failure_repeat_alert_hours``: how long an ``alerted`` incident stays silent before one reminder ping. ``0`` (or negative) re-alerts on every failing run (the pre-gate behaviour).r_cron_config_numberfailure_repeat_alert_hours) cron.jobsr"DEFAULT_FAILURE_REPEAT_ALERT_HOURSrrs r(_failure_repeat_alert_hoursrOs. ;=_af ggrfincidentct}|dkry|jd}|sy ddlm}|t j t |}t|z t|kS#ttf$rYywxYw)zAn ``alerted`` incident withholds the per-run ping until the reminder cooldown has elapsed since its last ping. A missing/unparseable ``alerted_at`` (ledger written before the column existed) delivers rather than swallowing an alert.rF alerted_at _ensure_aware)hours) rrHrrr fromisoformatrk TypeErrorr/ _hermes_nowr)rrrrlasts r(_repeat_alert_withheldrWs} ( )E zl+J +X33C ODE =4 )%"8 88 z "s*A**A<;A< output_filercD ddlm}m}||dt|xsd|j d|\}}||}|r|j dnd}|d k(xs|d k(xr t |} | |fS#t $r$} tjd |d| Yd} ~ y d} ~ wwxYw) aRecord a durable failure incident (grouped by job + error signature). Returns ``(withheld, incident_id)``; withheld=True when the signature's incident is already ``closed`` (operator ack) or ``alerted`` inside the ``cron.failure_repeat_alert_hours`` cooldown (a ping already went out) -> suppress the per-run ping. Store errors log at debug; the caller delivers as if none existed.r) get_incidentupsert_incidentrGr+rF)rrstateNclosedalerted?Incident store unavailable for job %s (delivery unaffected): %s)FN) cron.incidentsrrrkrHrr#rdebug) r<rrrr incident_id_is_newrrwithheldexcs r(_upsert_incident_for_failurerjs @. Is5;B'#''&/{ \ W ,)1 W%tH$a));)`@VW_@`$$  M Is  sA/A22 B;BBc ddlm}||dy#t$r$}tj d|d|Yd}~yd}~wwxYw)zBest-effort: a successful run marks the job's open incidents ``resolved`` (never touches an operator ``closed`` ack). Store errors log at debug; delivery is unaffected.r)!close_incidents_for_recovered_jobrGrN)rrr#rr)r<rrs r($_resolve_incidents_for_recovered_jobrsEhD)#d)4 h VX[\`Xacfgghs A<Arc|sy ddlm}||dy#t$r!}tj d||Yd}~yd}~wwxYw)zYBest-effort: mark incident ``alerted`` (no-op for closed; never resurrects an acked one).Nr)set_incident_staterz&Failed marking incident %s alerted: %s)rrr#rr)rrrs r(_mark_incident_alertedrsA Q5; 2 Q ={CPPQs ?:?ceZdZdZy)CronPromptInjectionBlockeda&Raised by _build_job_prompt when the assembled prompt (incl. runtime-loaded skill content, unseen by create-time scanning) trips the injection scanner; run_job turns it into a clean "job blocked" delivery. Assembled-prompt scanning (including loaded skill content) plugs the gap from #3968: create-time scanning only covers the user-supplied prompt field; skill content loaded at runtime was never scanned, so a malicious skill could carry an injection payload that reached the non-interactive (auto-approve) cron agent. Nrgrhrirjrfr(rrsrfrr9cJ|xsijdxsi}|jdrddg}ngd}|xsijdxsi}ddlm}||jd }|D]4}t|j }|s||vs$|j |6|S) uuToolsets a cron-spawned agent must never receive: ``messaging``/``clarify`` always (interactive); ``cronjob`` by default (loop prevention, not a security boundary — ``cron.allow_agent_scheduling: true`` lifts only that); ``agent.disabled_toolsets`` layered on top so per-job ``enabled_toolsets`` cannot widen past config.yaml's denylist. See #25752. r@allow_agent_scheduling messagingclarify)cronjobrragentr)parse_config_string_listdisabled_toolsets)rHagent.skill_utilsrrkr-append)r9cron_cfgdisabled agent_cfgr user_disabledrFs r(_resolve_cron_disabled_toolsetsrs rv&,"H||,-+6(.BI:,Y]];N-OPM"4y  D( OOD !" Orfper_jobc|Dcgc] }|dk7s | }}d|vr|Sddlm}||}t||zr|St|D]}||vs|j ||Scc}w)a Layer enabled MCP servers onto a per-job ``enabled_toolsets`` allowlist (else a per-job list silently drops every MCP server). Mirrors ``_get_platform_tools``: ``no_mcp`` sentinel -> none (stripped); any MCP server already listed -> allowlist, add nothing; else union all enabled.no_mcpr)enabled_mcp_server_names)hermes_cli.tools_configrsetsortedr)rr9tr$r enabled_mcprFs r( _merge_mcp_into_per_job_toolsetsrs|! 2AAMa 2F 27 @*3/K 6{[  {#  v  MM$   M3s A"A"c|jd}|rtt||xsiS ddlm}t ||xsidS#t $r}td||d}~wwxYw)u+Toolset list for a cron job. Precedence: per-job ``enabled_toolsets`` (+ MCP merge) > ``cron`` platform config (``_get_platform_tools``, which strips _DEFAULT_OFF_TOOLSETS so fresh installs run without ``moa``). A lookup failure fails CLOSED: the run errors out. 1. Per-job ``enabled_toolsets`` (set via ``cronjob`` tool on create/update). Keeps the agent's job-scoped toolset override intact — #6130. Enabled MCP servers are layered on per ``_merge_mcp_into_per_job_toolsets`` so a native-toolset allowlist does not silently strip MCP tools. 2. Mirrors gateway behavior (``_get_platform_tools(cfg, platform_key)``) so users can gate cron toolsets globally without recreating every job. 3. Never ``None``: AIAgent reads ``None`` as "every toolset", so an unreadable ``platform_toolsets.cron`` restriction would hand an unattended job the full default set (#111380). The raise reaches ``run_job``'s failure path, which records the error on the job and opens an incident, so the operator sees it instead of a widened run. enabled_toolsetsr)_get_platform_toolsr@zCron toolset resolution failed, so this run was refused rather than given every tool. Check `platform_toolsets.cron` in config.yaml (`hermes cron doctor`): N)rHrlistrrrr# RuntimeError)r<r9rrrs r(_resolve_cron_enabled_toolsetsrs~gg()G/W sybII?)#)V<==  [[^Z_ a  sA A'A""A'modelc Vddlm}m}|jd}|k||}|)tj d|jdd||Stj d|jdd||jdd|t|tr|nit|S)aEffective reasoning config for a cron run. A per-job ``reasoning_effort`` pin beats global and per-model config and is model-independent by design (also governs an auth-fallback swap); clamping stays with provider transports. An unparseable pin warns and falls back to config.r)parse_reasoning_effortresolve_reasoning_configreasoning_effortz-Job '%s': using per-job reasoning_effort '%s'rG?uJob '%s': invalid stored reasoning_effort %r — ignoring the pin and falling back to config resolution. Fix with `cronjob action=update job_id=%s reasoning_effort=` (valid: none, minimal, low, medium, high, xhigh, max, ultra).) hermes_constantsrrrHrinfowarningrJrKrk)r<r9rrrpinnedparseds r(_resolve_job_reasoning_configrsR WW' (F '/   KKGQUWZI[]c dM > GGD#   GGD#   $:c4+@Cb#e* UUrf)_ensure_cron_diradvance_next_runsclaim_dispatchclaim_job_for_firefire_claim_fenceclear_run_claim get_due_jobsheartbeat_fire_claimheartbeat_run_claim mark_job_runsave_job_outputself_removal_delivery_allowedself_removal_delivery_scopeuse_cron_store)_TERMINAL_STATESHANDOFF_ADOPTION_GRACE_SECONDScreate_executionfinish_execution get_executionmark_execution_handoff_pendingmark_execution_runningrecover_interrupted_executionsz[SILENT]z[CRON_FAILURE]rc|xsdj}|r|djtk7rydj|ddj }|xsdS)a Return failure evidence when an agent response declares a cron failure. Only the exact, standalone first line is control text. The caller keeps the complete response in the saved run output while routing this evidence through normal failure bookkeeping. r+rN rEzCron agent reported failure.) splitlinesrCRON_FAILURE_MARKERjoinr-)rlinesevidences r(_cron_failure_marker_errorrsX ZR # # %E E!HOO%)<<yyqr#))+H  555rfcddlm}||S)uvTrue when a cron final response should suppress delivery: ``[SILENT]`` (or SILENT / NO_REPLY / NO REPLY) as the whole response OR its own first/last line — NOT mid-sentence. Shares the webhook-lane matcher in :mod:`gateway.response_filters` so the two cannot drift. Recognizes the bracketed ``[SILENT]`` sentinel (whole-response, first line, or last line) plus the bracketless ``SILENT`` / ``NO_REPLY`` / ``NO REPLY`` variants the model emits when it drops the brackets (#51438, #46917). Whitespace-trimmed and case-insensitive. A token buried mid-sentence is treated as real content and delivered. r)is_autonomous_silence_response)gateway.response_filtersr!)rr!s r(_is_cron_silence_responser#sH )$ //rf_parallel_pools_parallel_pool_max_workersrrcJ| t||fStt|fS)u``(home key, job id)`` — the identity of one in-flight cron run. ONE gateway process ticks every profile, so a job id alone is not unique: two profiles routinely carry identically named jobs (``daily-brief``) and sharing a key made the second profile's run read as a duplicate of the first and get skipped. ``home`` defaults to the active cron scope's home, which the ticker binds per profile for the whole tick. )rr})rrs r( _inflight_keyr'5s.&*%5OD !`f gg?K[K];^`f ggrf_inflight_home_pathsc*|tt|<|S)zMRecord ``home`` under its key so a claim can be mapped back to a usable path.)r(rrs r(_remember_inflight_homer+Fs26./ Krfhome_keycFtj|xs t|S)zOReal home Path for an in-flight key's home half (falls back to the key itself).)r(rHr)r,s r(_inflight_home_pathr.Ls  # #H - ?h?rf_running_job_ids_running_fire_owners_restart_safe_waiter_job_ids_running_worker_pids_running_since_running_allowance_s_running_futures_forced_release_count_forced_releasesg>@_interrupted_job_idsc$eZdZdZdefdZddZy)_CancelEventLikezYStructural type for cancellation sources (``threading.Event``, ``_CombinedCancelEvent``).rcyNrrds r(is_setz_CancelEventLike.is_setrfNcyr=rr>s r(rz_CancelEventLike.setr@rfrN)rgrhrirjboolr?rrrfr(r;r;sc!!rfr;c:eZdZdZdedddfdZdefdZd dZy) _CombinedCancelEventzDuck-typed ``threading.Event`` ORing several cancellation sources (fire-claim heartbeat ``lost_ownership`` + per-transport events). Workers only call is_set()/set(), so no pump thread. eventsr;rNc>|Dcgc]}|| c}|_ycc}wr=)_events)rdrFevents r(rcz_CombinedCancelEvent.__init__s+1G%U5FG Gsc:td|jDS)Nc3<K|]}|jywr=)r?).0rIs r( z._CombinedCancelEvent.is_set..ss r(r?z_CombinedCancelEvent.is_sets.sZCQZN) _running_lock frozensetr/r0keysrrfr(get_running_job_idsrYs> [Z+;>R>W>W>Y+YZZ[[[s +<Ac @tj}t5tttj zDcgc]<}|d|t vrt|t |z dndtj|d>c}cdddScc}w#1swYyxYw)zPer in-flight job: ``{"job_id", "elapsed_s", "worker_pid"}`` (``worker_pid`` None for in-process runs). The drain wait publishes this so ``hermes update`` can say WHICH job it is waiting on.rEN)r elapsed_s worker_pid) rrVrr/r0rXr3roundr2rH)rrTs r(get_running_job_detailsr^s ))+C  .1E1J1J1LLM  1vADAV%nS&9 91=\`/33C8 :      s'BABBBBc tj}t5tjDcic]\}}|tvs|||z }}}|Dcic]}|t vs |t |c} ddds t Stdz |Dcgc] }| vs| }}|ri}tjt5ddl m }|Dcic]}|jd|}}dddtt}t5|D]\} } |d|k(r|j|dnd} t!| xsi} | rt#| d| zdz} | |<|tvsT| t |<^ dddt fd|jDScc}}wcc}w#1swY9xYwcc}wcc}w#1swYxYw#1swYZxYw) uIn-flight job IDs older than their stale-inflight allowance (``max(2 * interval, cron.inflight_max_minutes)``) — the scheduler's own definition of a claim that can no longer be making progress. ``sweep_stale_inflight`` cannot release these while the worker thread is still alive (a delivery blocked on a dead transport, #115469), so the gateway restart drain reads this to skip them the way it skips wedged chat turns; restart is their remedy. NN@r load_jobsrGrE@c3TK|]\}}|t|k\s|d!ywrR)max)rLrTage allowances floor_secondss r(rMz%get_wedged_job_ids..s2[3sc*S/=6Y/YA[s( ()rrVr3itemsr/r4rW_inflight_min_allowance_minutesr!r"r#rrbrHrr}_job_interval_minutesre)rrTstartedages unresolvedby_idrbj local_home allowance local_jobinterval_minutesrgrhs @@r(get_wedged_job_idsrus ))+C e7E7K7K7Mi|sGQTXhQhS7]"ii@DdOcHcc/44d e {35#?J?    + : +-6[9QUU4[!^9E9 :%%5%78  :! :) 14Q:1EEIIc!f-4 #8b#I # #Is5E/E/L MI"+ 3**09(- : : [#zz|[ [[5jdee @: : : : :sqF)FFF) F$# F$/F) F6!F6GF;-GAG ' G  F))F3;GG  Gcvt||}t5|tvxs|tvcdddS#1swYyxYw)aTrue when THIS process has an in-flight run of ``job_id`` FOR ``home`` (default: the active cron scope's home). The bare-id accessors above report the host-wide union so the shutdown drain and metrics see every profile's work. Liveness consumers must not: one process ticks every profile, so a ``daily-brief`` running in profile A would otherwise report profile B's idle ``daily-brief`` as running and keep B's stale one-shot alive as a "possibly live run". N)r'rVr/r0rrrTs r(is_job_runningrxs@  %C F&&E#1E*EFFFs/8cddlm}t|tt }|j 5}t 5|r|tvr ddddddytj|tjt|<tt|< ddddddy#1swYnxYwdddy#1swYyxYw)aAtomically add ``job_id`` to the in-flight set; False (caller must skip) if already mid-run. Single dedupe owner for ticker + manual runs (the fire claim's 300s TTL is outlived by real jobs). Callers MUST pair success with ``release_running_job`` in a ``finally``. This is the single dedupe owner shared by the ticker's ``_submit_with_guard`` and manual runs (``tools/cronjob_tools``): the fire claim alone cannot prevent a double-fire because its TTL (300s) is routinely outlived by real jobs, after which a manual ``cronjob(action='run')`` would claim successfully and run the same job concurrently (idea from #53395 by @izumi0uu). Registration also makes the run visible to ``get_running_job_ids`` (the gateway shutdown drain, #60432) and ``mark_running_jobs_interrupted``. Dedupe is per PROFILE: the key carries the active cron scope's home, so one multiplexing process never treats two profiles' same-named jobs as one. r) retirementNFT) hermes_cli.backend_retirementrzr'r+r}workrVr/addrr3_FUTURE_PENDINGr5)rrzrTadmitteds r(try_register_running_jobrs9  78H8J K LC  h 3"22 S!#iiks /s.B< B'B<>B'B<'B0 ,B<<Cc,t||}t5tj|tj |dt j |dtj |dtj |ddddy#1swYyxYw)aRemove ``job_id`` from the in-flight running set (idempotent). ``home`` MUST be passed by any caller that does not run inside the same cron scope the claim was registered under. The scope is a ContextVar the ticker binds per profile: a pool worker releasing in a ``finally`` outside ``ctx.run`` resolves the default (launch) home instead, so the discard misses the real key and every secondary profile's claim leaks. N) r'rVr/discardr3popr4r5r2rws r(release_running_jobrsv  %C ,  %3%  d+S$'  d+ ,,,s A.B  Bctjt5txsi}t |t r|j dinij d}|t|}|dkDr |cdddSdddtdj}|r t|}|dkDr|S tStS#1swYDxYw#ttf$r#tjd|tYtSwxYw)zZStale allowance floor (min): ``cron.inflight_max_minutes``, else env escape hatch/default.r@inflight_max_minutesNr HERMES_CRON_INFLIGHT_MAX_MINUTESz=Invalid HERMES_CRON_INFLIGHT_MAX_MINUTES=%r; using default %s)r!r"r#rrJrKrHrrr-r/rrr_INFLIGHT_MIN_ALLOWANCE_MINUTES)_ucfg_cfg_valvalraws r(rjrjs   Y ' #%/t% D D FC  1*CQw  +* **)I& 1 NNO/ 1 +*  1sAB<C<C*C:9C:_cron_interval_cacheexprc|tvr t|Sd}tjt5ddlm}|rgddlm}ddlm}|j}|||}|j|}|j|}||z jdz } | dkDr| nd}ddd|t|<|S#1swYxYw)zYCron expression cadence (gap between next two fires) in minutes; None -> floor allowance.Nr)_ensure_croniter)croniter)rr`) rr!r"r#rrrrrget_next total_seconds) rr$r _croniterrbaseitfirstsecondgaps r(_cron_interval_minutesr-s ###D)) F   Y ' ..   7 )8<<>D4&BKK)E[[*FE>002T9C!GSF ."( M . .s A5B55B>ctjt5|jd}t |t r"|j rddlm}||xsi}t |trt|jd}|dk(r)|jd}|r t|ndcdddS|dk(r0tt |jd xsd cdddSdddy#1swYyxYw) zBest-effort job interval in minutes (None if unknown / one-shot -> floor). ``schedule`` is persisted as a parsed dict; the string path is only a fallback for programmatic callers.r>r)parse_scheduler?rAminutesNr@rr+) r!r"r#rHrJrkr-rrrKrr)r<r>rr?rs r(rkrkCs   Y ' O77:& h $)9 0%h/52H h %<<'Dz!",,y1)0uW~d O Ov~-c(,,v2F2L".MN O O O  O sB C+-+C++C4c 2tj}t5tdtDtj Dcic]\}}|dt ||z dc}}tttdcdddScc}}w#1swYyxYw)zZProbe-visible snapshot; non-zero ``forced_releases`` means a job wedged and was recovered.c3&K|] }|d ywrRrrSs r(rMz+get_inflight_guard_stats..[sAc!fArUrE)runningrunning_ages_secondsforced_releasesrecent_forced_releasesN) rrVrr/r3rir]r6rr7)rrTrls r(get_inflight_guard_statsrVs ))+C >A0@AA%3$8$8$:% CAcGmQ//% 5&*+;&<>>>%>>s-B B %B B  BrF age_secondsallowance_secondsc||t|dt|dtjd}t5tj |tdt =ddd tdz dz }t|jt|dd5}|jtj|d zdddy#1swYnxYw#1swYyxYw#t$r }tj!d |Yd}~yd}~wwxYw) z@Persist a countable signal for one forced release (best-effort).rE)rrFrrrNr@zinflight_forced_releases.jsonlautf-8encodingrz*Could not append forced-release record: %s)r]r isoformatrVr7r_FORCED_RELEASE_HISTORYr}rparentopenwritejsondumpsr#rr)rrFrrentrypathfhes r(_record_forced_releaserds[!,"#4a8m%%'  )E 8& 6666 78F!F*-MM% $g . /" HHTZZ&- . / / 88  / / F A1EEFsA$C &3C"(CC" CCC"C"" D +DD ctt}t5tDcic]#}|d|k(r|dtj |%}}dddj Dcgc]!\}}||tus|jr|#}}}|siS ddl m }||Scc}w#1swY]xYwcc}}w#t$ricYSwxYw)ubLatest durable execution per releasable-looking claim (missing/pending/done future), one indexed query so the healthy path pays no DB work. Snapshot under _running_lock — iterating the set while try_register/release mutate it raises RuntimeError. Scoped to the ACTIVE profile's claims: the executions ledger it queries is that home's. rrEN)latest_executions) rr}rVr/r5rHrir~donecron.executionsrr#)rqrT claim_futuresrfut candidates _latest_execss r((_latest_executions_for_releasable_claimsrxs!!1!34J  ( 3q6Z+? F$((- -   #0"5"5"763 ;#0CHHJ J  FZ((     s4 B)(B$ B)(&B5 B;$B))B2; C C row claim_startedc|jd}|sy ddlm}|tj|}|j |k\S#t ttf$rYywxYw)aTrue when the ledger row was claimed at/after this in-memory claim. An older terminal row is the PREVIOUS run's (try_register->create_execution window); releasing on it would double-dispatch. Unparseable timestamps fail closed (the age path still bounds the claim). claimed_atFrr) rHrrrr timestampr/rOSError)rrr_ensure_aware_tsrow_tss r(_row_belongs_to_claimrsb&J ?!("8"8"DE!]22  7 +s3A A! A!rfrrc |jdxs|}|turdn|dnd}tjd||||||t |||||dk(ry|jdxsi}t |t r)|jd tjd ||y t|d d |d z dd|d z ddy#t$r!} tjd|| Yd} ~ yd} ~ wwxYw)zWARNING log + probe record for one forced release, then ``last_error`` unless the ledger already holds the run's real outcome or the job has a finite repeat budget.rFpendingNmissingfinisheducron.inflight.forced_release event=forced_release reason=%s job='%s' id=%s age=%.0fs allowance=%.0fs future=%s — stale in-flight claim released; the job was skipping every fire with 'already running'ledger-terminalrepeattimesucron.inflight.forced_release.job_untouched job='%s' id=%s — finite-repeat job released without mark_job_run (repeat budget preserved); row left in place so it re-fires normallyFz+Stale in-flight claim force-released after <z.1fz m (allowance z=m); previous run never released the scheduler in-flight guardz.Could not record forced release for job %s: %s) rHr~rrrrJrKr r#) r<rrfrrrrrF future_staterrs r(_record_stale_releasers 776? $fD # 69IZdL NN K fc9l < 64i8""WWX  $"F&$FJJw$7$C D &   T E9#(3H#b.-., - . TGQRSSTs$C C+ C&&C+due_jobsc |xsgDcic]%}t|ts|jd|'}}tdz}t j}g}ddlm}t}tt}|jD cic]\} }| t|} } }t5tD cgc] } | d|k(s | c} D]:} | d} tj| }| |t| <*||z }| j| }|}|rt|d|zdz}t j| }|t"urn||j%s|j| }|"|jd|vrt'||rd }n ||k\rd }ntj)| tj+| dt,j+| dt j+| dt.dz a|j1| ||||f= ddd|D]+\} }}}}t3|j| xsi| ||||-|Dcgc]}|d c}Scc}wcc}} wcc} w#1swY\xYwcc}w) uForce-release in-flight claims that can no longer be making progress; returns released ids. Stale = older than ``max(2 * interval, floor)`` AND (no live future — submit path hung before ``pool.submit`` returned — or finished without discarding the id) — or the claim's OWN ledger row is terminal regardless of age. Each release logs WARNING ``event=forced_release``, bumps the probe counter, mirrors JSONL, and writes ``last_error``. rGr`r)rrENrcstatusrrf)rJrKrHrjrrrrrr}rirkrVr/r3rer5r~rrrrr4r6rr)rrprorhrstale_terminal_states_latest _local_homejid _intervalskrTrrlrfrtrrrlatestr_reasonss r(sweep_stale_inflightrss'/n" MAt9LQUU4[!^ ME M35CkkmLFC#,Q//LJL &@/G!1Q4;3FAG% @CVF$((-G&)s#-C)~~f5 %I 31A+AD+HI "&&s+Co%[[(F"JJx(,<<)&':* !  $ $S )   sD ) $ $S$ /  d + !Q & ! LL&#y#v> ?K% @&@P16],YWeii/52vsIsT[\] QAaD  k NMH&@&@T !s;III- I%6 I I EI% I1 I%%I.) only_ownersrc pt5tt}tj Dcgc]*\}}||vr!|j D]\}\}}||||f,}}}}}}||D cgc]} | dd| df|vs| }} |D  chc] \} }} } | } } }} } |"|j dt | z |z Dtjd|Ddddg}D]k\}}}}|d}|s(tjd||j|7 t|5t|d||r|j|dddm|Scc}}}}}wcc} wcc} } }} w#1swYxYw#1swY/xYw#t$r!}tjd ||Yd}~d}~wwxYw) aBest-effort: mark every in-flight cron job interrupted; returns the job IDs marked. Called by gateway shutdown right after ``process_registry.kill_all()``: a job whose tool was killed must never report success. ``only_owners`` (``(job_id, fire_owner)`` pairs) restricts marking. Tokens go into ``_interrupted_job_ids`` BEFORE ``last_status`` is written so ``run_one_job`` sees them. NrErpc3@K|]}d|dt|dfyw)Nr)r.rSs r(rMz0mark_running_jobs_interrupted..s+ sD"5c!f"=> sc32K|]\}}}}||n|ywr=r)rLtokenrT_owner _profile_homes r(rMz0mark_running_jobs_interrupted..%s*$ 1sFM&EC /$ szdJob '%s' interrupted before its durable fire owner was registered; leaving persisted state untouchedFexpected_fire_ownerz%Failed to mark job %s interrupted: %s)rVrr1r0riextendr/r9updaterrrrr r#)rrrestart_safe_waitersrT executionsrowner profile_home active_firesfire_t_o_pregistered_keysmarked_token fire_ownerrrs r(mark_running_jobs_interruptedrs  "#?@$8#=#=#?  Z..0:0@0@0B  -,|C - -   "-9bTd1gaj$q'=RVa=aDbLb6BCC?2sB3CC      %69MM   ##$ 5A$  + 2F1=O-ZQ NN4  MM& !  O - *E6zKMM&) *O& MW cC  N * * O NNBFA N N Osk*E3/E E3,E&E& E3E+ AE3% F 0!E?F E33E<?F F  F5F00F5rct|}t5||tvr dddy|tvcdddS#1swYyxYw)aNon-destructive peek: has shutdown marked THIS execution interrupted? Used before deciding what to deliver; does not clear the flag (the authoritative pre-``last_status`` check needs it). ``token`` scopes to one execution so a fresh run reusing the job ID isn't poisoned.NT)r'rVr9)rrrTs r(_is_interruptedr@sL  C +  *>!>++**+++s 88Act|}t5d}||tvrtj|d}|tvrtj|d}|cdddS#1swYyxYw)zReturn True and clear the flag if shutdown marked THIS execution interrupted. Called right before ``last_status`` is written; consuming stops the flag leaking into a later run.FNT)r'rVr9r)rrrThits r(_consume_interrupted_flagrKsk  C   *>!> ( ( /C & & ( ( -Cs AA  A)get_idle_secondslimit_spoll_sstop future_donec|j|s5|ry t|xsd}||k\ry|j|s5y#t$rd}Y&wxYw)uPoll idle time until limit (-> True), stop, or the future completes (-> False). Uses ``threading.Event.wait``, not asyncio, so a blocked event loop cannot disable the watchdog. Driven by ``threading.Event.wait`` (a kernel timeout), not asyncio, so a blocked event-loop / ``run_job`` thread cannot disable this watchdog the way ``asyncio.sleep`` / ``wait_for`` would (family A of #94285 — the 4118s-idle-on-a-600s-limit cron hang). Returns True when *limit_s* of inactivity was observed. FrT)waitrr#)rrrrridles r(_inactivity_watchdog_looprZseii = )+2s3D 7?ii   D sA AActdj}|sy t|S#ttf$rt j d|YywxYw)zParse HERMES_CRON_TIMEOUT (seconds). 0 = unlimited; bad input = 600. Shared by the inactivity monitor and the cwd-lock bound so they can't drift: the lock bound must stay >= the inactivity limit or waiters fail while a healthy holder runs.HERMES_CRON_TIMEOUTg@z2Invalid HERMES_CRON_TIMEOUT=%r; using default 600s)rr-rr/rrr)rs r(_cron_inactivity_secondsr rsS 0 1 7 7 9C Sz  "KSQs )%AA max_workersc,ttt}tj |}|t j ||k7rH||j ddtjj|d}|t|<|t |<|S)a]Return (or create) the persistent parallel pool for the ACTIVE profile. Pools are per profile home because ``cron.max_parallel_jobs`` is a per-profile config key: a single process-global pool is created by whichever profile the multiplexing gateway ticks first and then silently imposes that profile's limit on every other profile. Frcancel_futuresz cron-parallel)r thread_name_prefix) rr+r}r$rHr%shutdown concurrentfuturesThreadPoolExecutor)r rpools r(_get_parallel_poolrs 23C3EF GD   t $D |155d;{J   MMuUM ;!!44#5I $+6"4( Krfct|D]D}tj|d}tj|d|2|j ddFy)uDrop the parallel pools of homes this process no longer ticks. Pools are per home and used to live until ``atexit``: a host that serves many profiles — or churns them — accumulated one ThreadPoolExecutor and its live ``cron-parallel`` threads per home ever ticked, and a deleted profile's pool was never reclaimed. ``wait=False`` so the ticker is never blocked; queued work still drains before the executor dies. NFr)rr$rr%r) home_keysrTrs r(discard_parallel_poolsrsOI<""3-"&&sD1   MMuUM ; z&drain_delivery_queue..&s  * @ rf)cron.delivery_queuerRrSexists)rVrWrRrSs`` r(drain_delivery_queuer`s)0 7>>    rfir`rC?cd} ddlm}|xsi}t|tr|j dini}|j d}|t |}|dk\r|S|S#t $r!}tjd|Yd}~|Sd}~wwxYw)z6Return the wall-clock bound for cron post-run cleanup.g$@r)rr@cleanup_timeout_secondsNz3Failed to load cron cleanup timeout from config: %s) hermes_cli.configrrJrKrHrr#rr)defaultrr9r configuredtimeoutrs r(_cron_cleanup_timeout_secondsrh9sG Q1m!r*4S$*?37762&R\\";<  !J'G!| N Q JCPP NQsAA B )BB )timeout_secondslabelric | tn t|}|dkr yt jg d  fd }t jt|d|d}|jj|stjd |||y rtj d|| dyy#ttf$r"}tj d|||Yd}~yd}~wwxYw) zDRun fallible post-run cleanup without permanently wedging a cron ID.NrTzJob '%s': %s failed: %sFc jy#t$r}j|Yd}~0d}~wwxYw#jwxYwr=) BaseExceptionrr)rcleanuprrs r(_runnerz/_run_cron_cleanup_with_timeout.._runner[sE  I HHJ  LL    HHJs  ?:A?AAz cron-cleanup-targetrFdaemonzRJob '%s': %s exceeded %.1fs; abandoning cleanup so future runs remain dispatchablerB) rhrr#KeyboardInterruptrr threadingEventThreadrstartrr) rnrrjrirgrroworkerrrs ` @@r(_run_cron_cleanup_with_timeoutryKs3B2I,.uUdOeG!|  I ?? D!#E  !-x(@OF LLN 99W  `        .uQxH =,-  LL2FE3 G sCC5C00C5c"eZdZdZdefdZdZy)_BoundedCronSessionDBzProxy SessionDB cleanup calls through the cron cleanup timeout; after the first failure or timeout all later calls fail immediately (a damaged connection leaks at most one worker).rc.||_||_d|_y)NF) _session_db_job_id _disabled)rdr1rs r(rcz_BoundedCronSessionDB.__init__ys% rfc`tjtsSfd}|S)Ncjr tdifd}t|jdd}|s+j d}||d_t dd j d S) Nz9session finalization disabled after prior cleanup failurecL id<y#t$r }|d<d}~wwxYw)Nvaluer)rm)rargskwargsr$rqs r(_callzB_BoundedCronSessionDB.__getattr__.._bounded.._calls7&,d&=f&=F7O$&)F7Os  ##zsession finalization (ra)rrjrTzsession finalization method z timed outr)rrryr~rH TimeoutError) rrrokrr$rFrdrqs `` @r(_boundedz3_BoundedCronSessionDB.__getattr__.._boundeds~~"#^__F 0dll4J4&PQ2RTB 7+$K!%"%A$z#RSS::g& &rf)r0r}callable)rdrFrrqs`` @r( __getattr__z!_BoundedCronSessionDB.__getattr__~s.))40M '0rfN)rgrhrirjrkrcrrrfr(r{r{usa3 rfr{rnow_isomodec  d|d|d|d|d S)zKCommon markdown header for the short-circuit run docs (no_agent / monitor). # Cron Job:  **Job ID:**  **Run Time:** z **Mode:** rr)rrrrs r(_job_doc_headerrs5 xj!h "F" rfc|jdxsdjxsd}|r1t|jstj d||y|S)zLConfigured job workdir, or None when unset / no longer a directory (logged).workdirr+NuGJob '%s': configured workdir %r no longer exists — running without it)rHr-ris_dirrr)r<rrs r(_resolve_job_workdirrsSwwy!'R..08DGtG}++- U G  Nrfc ddlm}|t|j d}t|xsdjsdd l m }t|||St||} t|||| \}} tj!d} t#||| d} |sd|d| d| } d | d| d| | fSt%| s#tj'd|d| dt(dfS| js#tj'd|d| dt(dfSd| d| d| dfS#t$rtj d|dY1wxYw#t$r(} tjd |d d | } }Yd} ~ d} ~ wwxYw)uno_agent short-circuit — the script IS the job (no AIAgent, no tokens). stdout → delivered verbatim; empty stdout or wakeAgent=false → silent success; non-zero exit/timeout → error alert. r)load_hermes_dotenvr6z%Job '%s': no_agent .env reload failedTexc_infoscriptr+)NO_AGENT_WITHOUT_SCRIPT_ERRORr cancel_eventz.Job '%s': script execution raised unexpectedlyFzScript execution failed: N%Y-%m-%d %H:%M:%Szno_agent (script)u⚠ Cron watchdog 'z' script failed  Time: z**Status:** script failed ru8Job '%s' (no_agent): wakeAgent=false gate — silent runz%**Status:** silent (wakeAgent=false) u0Job '%s' (no_agent): empty stdout — silent runz"**Status:** silent (empty output) z --- )hermes_cli.env_loaderrr}r#rrrHrkr-rr_block_and_pause_jobr$_run_job_script_with_claim_heartbeat exceptionrr&r_parse_wake_gater SILENT_MARKER)r<rrrr script_pathr _job_workdirroutputrrheaderalerts r(_run_no_agent_jobrsU<'7'9:''(#K { b ! ' ' );#FH6STT(V4L>9 lO F m$$%89G Xvw8K LF "(+>hI    =fXRH%QWWW F # NPVWxEF W[[[ <<> FOxBC]TXXX F89VHB/ ==Q U I6R7u=F>s)D&'E&!E  E  E?E::E? extra_promptcddlm}m}||sd|dfS||}tj d}t |||d}|j sYtjd||jd|d|jd |} d |d |jd | |jf|dfS|js&tjd |d|dtdf|dfSd||jfS)anMonitor gate (hash-suppressed change detection). Must run BEFORE any agent machinery so an unchanged tick costs no LLM/delivery. Returns ``(early_result | None, extra_prompt, monitor_context)``. Monitor context is runtime data and must remain distinct from a user-authored ``extra_prompt`` so the prompt scanner keeps its strict user-input boundary. r) check_monitorjob_has_monitorNrmonitorz#Job '%s': monitor source failed: %su⚠ Cron monitor 'z' source failed rFz#**Status:** monitor source failed ru<Job '%s': monitor output unchanged — suppressing agent runTz-**Status:** no_change (agent run suppressed) ) cron.monitorrrrr&rrrrchangedrr context_block) r<rrrrr_mon_mon_nowr _mon_alerts r(_apply_monitor_gaters< 3 \4''  D}%%&9:H Xvx CF 77  :FDJJO  *=zzlJ   fXB4::,bQS]_c_i_i   << RTZ[ VHJK]\`   t11 11rfc:eZdZUdZeed<eed<eed<eed<y)_CronJobConfigz4Config-derived inputs for one agent-backed cron run.r9r model_cfgcron_default_providerN)rgrhrirjrK__annotations__rkrrrfr(rrs> I JNrfrc 0|jdxstdxsd}d}i}i} ddlm}t t dz }t jj|r7|t|}|jdxsi}|jdxsi} d} t| trXt | jdxsdj} t | jdxsdj}|jds| r| }n|t|tr|nHt|tr7|jd xs&|jdxs|jd nd} t | xsdjxs|}t|tr|js0t!d |d|jddtdd|d t#j$t5ddlm} |jdi}t|tr|jdr | dd d d t+||||S#t$r!} tjd || Yd } ~ d } ~ wwxYw#1swYDxYw)aLoad config.yaml and resolve the run's model: per-job pin > cron.model (fleet default) > the main agent model (config ``model:``, then HERMES_MODEL). Re-read every tick (no cache) so ``hermes cron edit --model`` and ``hermes model`` both apply next tick.r HERMES_MODELr+r)load_user_config_effectivez config.yamlr@model_providerrerFz8Job '%s': failed to load config.yaml, using defaults: %sN Cron job 'z%' has no model configured (job.model=z, HERMES_MODEL=zY, config.yaml model.default missing or empty). Set a per-job model via `hermes cron edit z= --model ` or set a default with `hermes model `.)apply_ipv4_preferencenetwork force_ipv4T)force)rHrhermes_cli.config_effectiverrkr}osrr_rrJrKr-r#rrrr!r"rrr)r<rrr_cron_default_provider_cfg _model_cfgr _cfg_path_cron_cfg_for_model_cron_default_model_mainrr_net_cfgs r(_load_cron_job_configrsf GGG  F 0 @ FBEDJ^J(*]:; 77>>) $-d9o>D'*0bJ"&((6"2"8b "$ -t4&)*=*A*A'*J*Pb&Q&W&W&Y#),-@-D-DEU-V-\Z\)])c)c)e&777#&/E+5Z*EJ%j$7#y1fZ^^G5LfPZP^P^_ePf=?  ,224=E uc "u{{} #'''*-.,^<?@!"()2  2     Y '.:88Ir* h %(,,|*D ! - . $z3I JJ+ ^QSY[\]]^ ..s%E3IAJ  J (JJ  JcZt|jditr|jdini}tdxs&|jddxs|jdd}|syt |j }|j s t|z }|jsy t|dd5}tj|}dddttr|SdS#1swYxYw#t$r"}tjd |||Yd}~yd}~wwxYw) zVPrefill messages from env or config.yaml (top-level key canonical; agent.* is legacy).rHERMES_PREFILL_MESSAGES_FILEprefill_messages_filer+Nrrrz8Job '%s': failed to parse prefill messages file '%s': %s)rJrHrKrr expanduser is_absoluter}r_rrloadrr#rr)r9rr prefill_filepfpath_pfprefill_messagesrs r(_load_prefill_messagesrOs(23777B3G(N$TVI78 6 77*B / 6 ==0" 5  ,  * * ,F    !#f, ==? &# 0 .C#yy~  .#-.>#EO4O . . QSY[acdes63C?C3C?1C?3C<8C?? D*D%%D*cFd} t|rOt||}|sA|jdr0tjt 5ddlm}||ddd|syt|||S#1swYxYw#t $rtjd|dd}YAwxYw)a}Pre-dispatch config validation: refuse unrunnable jobs (missing key, unready skill, unconfigured delivery) BEFORE AIAgent is built. run_one_job keys off BLOCKED_CONFIG_MARKER to record blocked_config and alert once (`preflight_alerted` bit). Must run after the wake gate so silent ticks stay silent. Opt-out: `cron.preflight: false`. Returns failure tuple or None. Npreflight_alertedr)clear_preflight_alertedu7Job '%s': preflight validation errored — failing openTr) _cron_preflight_enabled_preflight_job_configrHr!r"r#rrrr_blocked_config_result)r<rrr9 _pf_reasonrs r(_preflight_or_blockrgsJ  "3 '.sC8J#''*=">((34A+F34  !&(J ??44  NPVae f s*AA:A.A:.A73A::#B B rc  tjd|||d} ddlm}||}|rt nt}d|d |d tjd d |d }d|d|d|fS#t$rtj d|dYbwxYw)zMThe ``blocked_config`` failure tuple for *_pf_reason*, alerting once per job.uZJob '%s' (ID: %s): BLOCKED by pre-dispatch config validation — %s (no LLM call was made)Fr)mark_preflight_alertedz2Job '%s': could not persist preflight alert markerTrrrrrz **Status:** BLOCKED (configuration) The pre-run configuration check found a problem, so the agent did not run (nothing was charged). **Reason:** z Hermes tries again at the next scheduled time and clears this state on the first healthy run; this alert is not repeated. Check with `hermes cron doctor`. Set `cron.preflight: false` in config.yaml to disable this check.r+r) rrrrr#rBLOCKED_CONFIG_SILENT_MARKERBLOCKED_CONFIG_MARKERrr&)rrralready_alertedrmarker blocked_docs r(rrs NNd&*&Ob408.= )BWF xj!h$//0CDEF"l#7 7 +rfXQzl#; ;; b I6\` absA))!B  B jcc ddlm}m}ddlm}|j }|j dxs|jxsd} ||d}|j dr|j d|d<|di||fS#t$r} t| |} t| } | s| st|| | tjd || rd nd | t|jD]v} t| t st#| j dxsd j%} t#| j d xsd j%}| r|sr ddlm}m}| |d}| j dr| d|d<|| }|r||d<|di|}|| ||d<tj-d||j d|ddlm}||xs7t|j0t r|j0j dnd ||j d||d<||fccYd} ~ S#t$r#}tj3d|| |Yd}~qd}~wwxYwt|| | d} ~ wwxYw)a\Resolve the runtime, walking the fallback chain on auth/transient-network errors. Returns ``(runtime, model)``; provider+model swap atomically (never swap only the provider while keeping a paid primary model). Provider precedence: per-job pin > cron.model_provider > persisted global config (None lets resolve_runtime_provider read it).r)resolve_runtime_providerformat_runtime_provider_error) AuthErrorrN) requested target_modelbase_urlexplicit_base_urlzCJob '%s': primary provider resolve failed (%s: %s), trying fallbackauthztransient networkr+r)effective_runtime_providerresolve_entry_api_keyexplicit_api_keyz*Job '%s': fallback resolved to %s model %s)pre_agent_fallback_notice_fallback_noticez Job '%s': fallback %s failed: %sr)hermes_cli.runtime_providerrrhermes_cli.authrrrHrr#rJ$_is_transient_provider_resolve_errorrrrrr9rKrkr-hermes_cli.fallback_configrrrrrr)r<rrrrrrrruntime_kwargs resolve_excis_authis_transient_netr fb_providerfb_modelrr fb_kwargs fb_api_keyruntimerfb_excs r(_resolve_job_runtimer sr A) HHE #Gr'?'?G4I6X#!  77: 25''*2EN. /'9.95@@ +X[)4? L+<[IJP[ [ Q gF+>  M(/ ^EeT*eii 39r:@@BK599W-34::I012?Y?'A&P # @GKK 3X? Q.Gi*R\\[_B`",,"2"2:">fh7;;z2H/>*+(( ^ ?V\]] ^= ^@8EFKWW+XsI3A33 I,=CI'CH%I'I,% I.I I' II''I,r c |t|jdxsdjj}|sy ddlm}||}|j r1tjd||t|j|S y#t$r"}tjd|||Yd}~yd}~wwxYw)Nrr+r) load_poolz@Job '%s': loaded credential pool for provider %s with %d entriesz3Job '%s': failed to load credential pool for %s: %s) rkrHr-ragent.credential_poolr has_credentialsrrrentriesr#r)r rruntime_providerr rrs r(_load_credential_poolrs7;;z28b9??AGGI  i3)*    ! KKR(#dlln*= ?K "  i JFTdfghh isAB B;B66B;c ddlm}|}|r!tjd|t |yy#t $r!}tj d||Yd}~yd}~wwxYw)zRegister MCP servers for the agent's tool registry. Idempotent across ticks; non-fatal so a broken MCP server never kills a working job.r)discover_mcp_toolsz"Job '%s': %d MCP tool(s) availablez3Job '%s': MCP initialization failed (non-fatal): %sN)tools.mcp_tool_discoveryrrrrr#r)rr _mcp_tools_mcp_excs r(_init_cron_mcp_toolsrsZ ` @')  KK )CL>2(\ + ,, 5 5s C++C4prompttask_id worker_statecNt}|dkDr|ndd|jd} t| txr| jddk(|jd} t| trt | jdxsd nd t j dfd } fd } tjjd } tj}| j|jj||||d<dtj d t"ffd dfd }tj$|dt ddd} |j'sj)}nTd} tjj+h\}}|r| j)}nrn| | Qj1| j/ddr t3t|ts$t5dt7|j8d||S#t,$r| j/ddwxYw#j1| j/ddwxYw)zRun ``agent.run_conversation`` on a worker thread under the inactivity (not wall-clock) watchdog: default 600s, override HERMES_CRON_TIMEOUT, 0 = unlimited.rNg@r>r?once run_claimbyr+rcjsytdrjdtdd)N interruptz"Cron fire claim ownership was lostrz'' lost its durable fire claim ownership)r?r2r?r)rrrsr(_abort_if_fire_claim_lostz;_run_agent_with_watchdog.._abort_if_fire_claim_lost`sH  |':':'<    !< OO@ AZz1XYZZrfcrsytj}|z tkry| ty#t$rt j ddYywxYw)Nexpected_ownerz$Job '%s': run_claim heartbeat failedTr)rr_RUN_CLAIM_HEARTBEAT_SECONDSr r#rr)_mono _is_oneshot_last_claim_heartbeat_run_claim_ownerrrs r(_heartbeat_run_claim_if_duez=_run_agent_with_watchdog.._heartbeat_run_claim_if_duegsf"2   ( (+G G  % Z 7G H Z LL?TXL Y Zs 9!AArEr)r8r&Fctdsy j}t|jddxsdS#t$rYywxYw)Nr)rr,)r2r)rrHr#)_actrs r( _idle_secondsz/_run_agent_with_watchdog.._idle_seconds~sRu45 --/D":C@GCH H  s.? A  A cFytjrdyy)N)rrrrrT)rr)_POLL_INTERVAL _cron_future_cron_inactivity_limitrL_inactivity_timeout _watch_stopsr(_watch_inactivityz3_run_agent_with_watchdog.._watch_inactivitys5 ! )  $*4JSa,*;*; =#'  =rfzcron-inactivity-Trprrz agent.run_conversation returned z instead of dict: rB)r rHrJrKrkrrrrrrrr r!run_conversationrtrurrvrwr$rr#rrr6rtyperg)rr7r<rrr8rr9 _cron_timeout _job_schedule _run_claimr@rI _cron_pool _cron_contextrS _watch_threadr$r_rNrOrPrLrQrFrGrHrRs` `` ` @@@@@@@@@r(_run_agent_with_watchdogr^Js -.M.;a.?]TNGGJ'M]D1Ym6G6G6OSY6YK%J:DZQU:Vs:>>$/526\^ NN,[ Z Z##6616EJ,,.M$$51167%LL!- X//#K5''$$ )9#f+bq/9J'KTXZM= ! -    ! ! )+,BV!((*FF$,,11<..1Ya-/)002F&)++- t<!%3IJ fd #.tF|/D/D.EEWX^Wa b   M t<  t.srfi cron_complete) interruptedremptycron_incomplete_no_outputu^Job '%s': session ended without a final assistant message (lifecycle=%s) — booking run as %sz5Job '%s': session lifecycle classification failed: %sFz#Job '%s': failed to end session: %srrz2Job '%s': failed to close SQLite session store: %s)r{get_compression_tipr#rsr!r"r0rrrsplitr-rr&r7session_lifecycle_statusesrHr end_session_end_session_on_closer%r )r1rrrrwr}_final_cron_session_id_compression_tipr_agent_session_id _title_base _cron_title _fallback _end_reason _statuses _lifecycler s r(_finalize_cron_sessionrs( F;K-X&::?K %5 " hhx~~/0"5;;=Q5AQ $ T+-*@*@*O)PQ &{4JKX #K1G5QWPXIY Z4"K Y::!? @ ; '|T B !):& ; ; ;  KVUVWW X, ( )  EvqQ KGK!" }< / .~ LhWb-.44 <Q& \ LL;VS I\(;se2FCPSHUW[[ [ \d &> KXW]_h 8*%!(#(]334GHIJ#$-+.T T {BI7==%>s<D8E98E6=.E1+E61E69 GAGGG)!HERMES_CRON_AUTO_DELIVER_PLATFORM HERMES_CRON_AUTO_DELIVER_CHAT_ID"HERMES_CRON_AUTO_DELIVER_THREAD_IDc:eZdZdZdededeefdZd dZd dZ y) _CronRunScopeu0Per-run ContextVar / tool-cwd scope for ``run_job`` (ContextVars, not os.environ, so parallel jobs don't clobber each other). Construct before the try, ``enter()`` as its first statement, ``exit()`` in the finally — every setter here has a matching reset there. HERMES_SESSION_* are deliberately NOT seeded from job["origin"]: it is delivery metadata, not a sender, and terminal/tts/skills/send_message tools would act as if the origin user were driving the agent. Delivery reads job["origin"] / HERMES_CRON_AUTO_DELIVER_* directly. r<r execution_idcddlm}m}ddlm}||_t |||_|dddd|jxsd|_tD]}||jdd|d|xs1|jd xstjj|_|jr||j|j|d |_d|_d|_y) Nr)set_session_vars_VAR_MAP)record_session_cwdr+F)platformchat_id chat_nameasync_deliverycwdzcron::rHERMES_CRON_SESSION)gateway.session_contextrrtools.terminal_toolr_var_maprr _ctx_tokens_CRON_DELIVERY_VARSrrHuuiduuid4hexr8_cron_session_var_cron_session_token_non_dispatcher_token)rdr<rrrrrrFs r(rcz_CronRunScope.__init__sF:  +C8 +! "! $( #D TN  r " #vha (c8O(cSWS]S]S_ScSc'de << t||T\\ :!)*?!@#' %)"rfNcb|jjd|_t|_y)N1)rrrrrr>s r(enterz_CronRunScope.enters+$(#9#9#=#=c#B &H%I"rfcZddlm}ddlm}||j||j |j %|jj|j |jt|jtD] }|j|jd"y)Nr)clear_session_vars)clear_session_cwdr+)rrrrr8rrrresetrrrrr)rdrrrFs r(exitz_CronRunScope.exits>9$,,'4++,  # # /  " " ( ()A)A B  % % 1 -d.H.H I' (D MM$  # #B ' (rfrB) rgrhrirjrKrkr rcrrrrfr(rrs1"*D"*#"*Xc]"*HJ (rfrcRddlm}m}ddlm}|t |t t |}|rk|dj|d|djt|d|d j|jd d n t|d y y ) z[Re-read .env for this run and publish the auto-deliver target into the session ContextVars.r)rreset_secret_source_cache)rrrrrrr thread_idNr+) rrrrrr}_resolve_delivery_targetrrkrH)r<rrrdelivery_targets r(*_reload_dotenv_and_publish_delivery_targetr sT0.01#3#56.s3O4599/*:UV3488_Y=W9XY56::!%%k2:BOT_D`@a rfceZdZUdZdZeeed<dZe ed<dZ e ed<dZ e ed<dZe ed<dZe ed <dZe ed <dZe ed <dZee ed <y) _CronAgentSetupz\Everything ``AIAgent(...)`` needs that is resolved from job + config (or a preflight block).Nblockedr+rr rr.reasoning_configfallback_modelcredential_poolfallback_notice)rgrhrirjrr _RunResultrrrkr rKrrr.rrrrrrfr(rr sif$(GXj !(E3OGT c NC c NCOS%)OXc])rfrc|j}t|j}t|||_ddlm}|jdijd}||jd}|||_t|t|||||_ |j|St|||\|_ |_|jjdd|_t!|t#|t$r|nit'|j|_t+|xsd|_t/|j||_t3|t5|r t7||}|rt9||||_ |S)zResolve model/runtime/reasoning/pool for the run, in the original gate order: exfil guard -> preflight (may block) -> runtime (+ fallback chain) -> credential pool -> MCP.)rr)resolve_turn_limitr max_turnsNr)r9rrrrrdrrHr._guard_job_credential_exfilrrr r rrrrJrKrkrrrrrrr_empty_requested_mcp_toolsetsr) r<rrrrsetup_resolve_turn_limit_mt _mcp_reasons r(_resolve_cron_agent_setupr" sL 66D "(( +E3D&AEL ((7B  # #K 0C {hh{#.s3E $'VXtDEM }}  !5c62!FEM5;!MM--.@$GE: Zd+TS5EE.d3;tE1%--HE t$3C> 268[QEM Lrfrrc \|j}|jdxsi}|d*id|jd|jdd|jdd|jdd|jdd|jdd|jdd |jd d |jd d |jd|jd|j d|j d|jd|jdd|jdd|jdd|jdd|jdxsijddt||dt|dd d!t| d"d d#d$d%d d&d'd(|d)|S)+Nprovider_routingrapi_keyrrrequested_providerapi_moderequest_overrides acp_commandcommandacp_argsrr.rrrrproviders_allowedonlyproviders_ignoredignoreproviders_orderorder provider_sortsortopenrouter_min_coding_score openroutermin_coding_scorerr quiet_modeTskip_context_filesload_soul_identity skip_memoryFskip_background_reviewrr@r2r1r) r rHrr.rrrrrrrC) rmr<rrrr2r1r prs r(_construct_cron_agentrJ smmG $ % +B  kk  I& Z( Z(   #;;';<   Z(  "++&9: KK * V$ ++ // // ++ -- &&.  &&*! "w# $ffVn% &&*XXl%;%Ar$F$FGY$Z' (8TB) *:$?+ ,- 0 $G},1 2 3 45 6 $7 89 :; <= rfc>eZdZdZdededefdZdedeedd fd Zy ) _FireAuditz[One usage_audit.jsonl line per fire (created once the agent exists; fire id + start clock).r<rrc|||c|_|_|_tjj |_tj|_ yr=) r<rrrrrfire_idrrt_start)rdr<rrs r(rcz_FireAudit.__init__r s9,/)$+tzzz|'' ~~' rfr$rrNctt|j|j|j d|j d|j dt |j d|j j d|jxsdttj|jz dz|d y)N prompt_tokenscompletion_tokens total_tokensresponse_silentdeliverr") tsrrrrrrdeliver_targetr duration_msr) r.r(rrrHrCr<rrIrrr)rdr$rs r(rz_FireAudit.writew s "kk||#ZZ8!',?!@"JJ~6#FJJ/@$AB"hhll95ZZ'4 04<< ?4GH  rf) rgrhrirjrKrkrcr rrrfr(rro s=e(D(#(c( D # 4 rfr)defer_agent_teardownrrrrrrc  |d}t|jdxs|jdxs|xsd}t|||||\}}||Sddlm} d|d t j d } tjd ||tjd |dd d} d} d} d}i}t|||} |j|jr!tjd||jt|t|||}|j}|j} t!||||}|j"w|j"ddlm}||jd| | ||| }|j)| r|st+| | ||| |s$|| |j-| SSt/| |SS|j} t1|} t3| ||||j| | } t5||| }t7| |||||j8||}t;|||| }|j<r7|j?r'tA|stC||j<d|}|r|nd}tE||||d|dz}tjd||jGtI|tA|xsddd||dfddlm}||jd| | ||| }|j)| r|st+| | ||| |s$|| |j-| SSt/| |SS#tJ$r9}tM|jNdt|}tjQd|| ddl)m*}||| rd|d<ddl+m,}||}|r||d <n%#tJ$rtj[d!|YnwxYw||jGi|dd"l.m/}tE||d#||||z}d$|d|fcYd}~ddlm}||jd| | ||| }|j)| r|st+| | ||| |s$|| |j-| SSt/| |SSd}~wwxYw#ddlm}||jd| | ||| }|j)| r|st+| | ||| |s$|| |j-| wwt/| |wwxYw)%uExecute a single cron job. Returns (success, full_output_doc, final_response, error). ``defer_agent_teardown``: if a list, the live agent is appended instead of torn down; the caller MUST call ``_teardown_cron_agent(agent)`` AFTER delivery (a torn-down async client can't deliver). ``extra_prompt``: per-fire context, never persisted. ``defer_agent_teardown``: when a caller passes a list, ``run_job`` skips the agent's async-resource teardown (``agent.close()`` + ``cleanup_stale_async_clients()``) in its ``finally`` block and instead appends the live agent to that list. The caller is then responsible for calling ``_teardown_cron_agent(agent)`` AFTER it has delivered the result. This closes the ordering window in #58720 where delivery ran against a torn-down async client (defense-in-depth alongside the interpreter-shutdown guard). When ``None`` (the default) teardown happens inline as before, so every existing caller is unchanged. ``extra_prompt``: optional per-run context from ``cronjob(action='run', prompt=...)`` (#57331). Appended to the stored prompt for this fire only — never persisted to the job definition. rGrFr7rNr)rmcron_r]z %Y%m%d_%H%M%SzRunning job '%s' (ID: %s)z Prompt: %sdr+z&Job '%s': using task-scoped workdir %s) defer_teardown_to_running_workerr&)rr2r1)r9rrfz ## Response rzJob '%s' completed successfully)rTz: zJob '%s' failed: %s)is_model_unreachable_failure_model_unreachable)hold_seconds_from_failure_quota_hold_secondsz3Job '%s': unreachable-failure classification failed)format_run_errorz (FAILED)F)0rkrHr run_agentrmrr&rrrrrrrr9rrrcron.scheduler_detached_workerr rrr_teardown_cron_agentr'rrr^r8rvrr-r#rrrrKr#rVrgrcron.unreachable_retryr cron.quota_holdr rcron.scheduler_diagnosticsr) r<rrrrrrearlyr7rm_cron_session_idrrr}_audit _worker_statescoperrrr _worker_teardown_deferredr$ralogged_responserr error_msgr r _hold_srs r(run_jobr sU&YF3776?Ocggh&7O6OZPH'VX|\ZME6  !vha (>(>(O'PQ KK+Xv> KK fTcl+ E EK#'FM #v| 4ET4 == KK@&%-- X237 "3 9vv)#vxD == $==n T$D   h 'eVXO_%a! 8 ";vxIY Z)#/$(//6%%UF3 )G ,C0 % S$u}}IY"$C/) 63%--&(5VVXwW  ! !n&:&:&F!& 5 56d>:JKN,:.@Y h?OTcSddfBgg 5x@ T&2KNL`^`2abdhiV^T1> T$D   h 'eVXO_%a! 8 ";vxIY Z)#/$(//6%%UF3 )U ,Aw''(3q6(3 .)D X K+Au5,0() B/2G-4)* X LLNPV W X   LLY '? CH:Y!7 Hq! " fb)++ T$D   h 'eVXO_%a! 8 ";vxIY Z)#/$(//6%%UF3 )U,: T$D   h 'eVXO_%a! 8 ";vxIY Z)#/$(//6%%UF3 )s]>BL6._cleanup_agent s U   \ J ' ) ,- U LLH&RS T T U  \ LLOQWYZ [ [ \s+$ AAAA B A<<Bzagent resource teardown)rrjrirB)ry)rrrir%s`` r(rr s \#v-F')rfc jd}t|trt|jdxsdnd s|dStjdxsdt j t j dtddffd } t  }|durtjd|dy d fd }t|dfd}| |dy | j|jdS#t$r#tjd d |d Yy wxYw# j|jdwxYw)zDRun ``run`` while keeping this job's owned durable fire claim fresh. fire_claimr=r+NrGrrcjd}|sy t|d|y#t$rtj ddYywxYw)NrFsuccessrz8Job '%s': failed to close unstarted execution ledger rowTr)rHrr#rr)rrr<rs r(_finish_unstartedz9_run_with_fire_claim_heartbeat.._finish_unstarted* sQww~.    \5 F  NNJ   s&!A  A rBz.Job '%s': initial fire_claim validation failedTrzEFire claim ownership could not be validated before execution started.Fz@Job '%s': fire claim ownership was already lost before executionz3Fire claim ownership lost before execution started.ctj}jts t sjt r/jt s t rtj}fjtjdytj}jtsyy#t$rftjddtj|z tk\r-jtjdtYyYwxYw)NrBz;Job '%s': fire claim ownership lost; interrupting stale runz%Job '%s': fire_claim heartbeat failedTrzRJob '%s': fire_claim could not be renewed within %.1fs; interrupting uncertain run) rrrrDr r  _FIRE_CLAIM_MISS_CONFIRM_SECONDSrrrr#r#_FIRE_CLAIM_HEARTBEAT_GRACE_SECONDS)last_confirmedrlost_ownershiprrs r(_heartbeat_loopz7_run_with_fire_claim_heartbeat.._heartbeat_loopB s )))89 +F5I4V< yy!ABFZ"5G:)-)9 "&&(NNU !%!1#))89$  DfW[ \NN$~5:;#&&(NN5; = ; s$C6C<&C#CA*D>=D>zcron-fire-claim-heartbeatc4tjddS)Nz.Job '%s': could not start fire_claim heartbeatTrrr)rsr(r]z0_run_with_fire_claim_heartbeat..f s "  c*q GZ^_abn  c*s' D&<E&)EE$E9F)rVrWverboserrr9c jds(tddjd}|dd<td}tjjd|k(}|s t ry /jd} | rjdr t| jd } t| trt| jd xsd nd }ttj}t!d}t"5|xsd|ft$j'|i<ddd t)d5t+fdcdddt"5t$j|}|*|j-d|st$j-|ddddS#t $r} d| } tjd d| jd } t| trt| jd xsd nd } tdd | fi| rd| init|d | n#t|d | wxYwYd} ~ yd} ~ wwxYw#1swY8xYw#1swYSxYw#1swYnxYw t"5t$j|}|*|j-d|st$j-|ddddy#1swYyxYw#t"5t$j|}|*|j-d|st$j-|ddddw#1swYwxYwxYw)uRun ONE due job end-to-end: execute → save output → deliver → mark. Shared by the built-in ticker and external providers' ``fire_due``; does NOT decide due-ness or acquire the initial claim (callers use the store CAS) but keeps it alive. True if processed (a job failure is recorded via ``mark_job_run``), False only if processing raised. ``cancel_event``: optional transport-level cancel (dashboard drain).rrGdirect_scheduled_instantsourcescheduled_instant_HERMES_CRON_EXTERNAL_WORKERTz*Restart-safe cron worker dispatch failed: z Job '%s': %sr'r=r+Frr)Nmanual_run_prompt manual_run_atc *t|S)NrVrWr9r claim_losttransport_cancelexecution_token)_run_one_job_body)r0rVrrGrr<rWr9s r(r]zrun_one_job.. s%'8%#!--%1$3(5rf)rHrrkrenviron_launch_external_cron_workerr#rrrJrKr robjectr}resolver'rVr0 setdefaultrr8r)r<rVrWr9rr executionrexternal_owner handoff_errorrr5r_stampedrr _fire_keyrrGs`````` @r( run_one_jobrSs s, 77> "$ Ih#''BV:WY 'oNs>*+LZZ^^$BC|SN  +C01"77./ 0x=L GGL !E/9%/FUYYt_*+BJhO#%--/Lc$i(I .  $ K.'' 26G.> (T 3 6155 6 6 6 >-11) >W @PE LLTE :GGL)E2 > 6 6 6 >-11) > >] >-11) > >s4 G) I?L J5 LAJ  I<A I79II7I//I77I<?J  J J!L+AK66K?M  AM M M M z3Interrupted by shutdown before terminal completion.rc|3t||r&t|dt|t|dtyt|ddy)uBookkeeping after fire-claim ownership loss. A transport-level cancel (dashboard drain) is not a real loss — we still own the claim, so record the interruption via the owner-fenced terminal write instead of leaving fire_claim/last_status stale; otherwise discard.NrBFrr)z6Fire claim ownership lost; stale result was discarded.)r r _OWNERSHIP_LOST_INTERRUPTEDr)rrrs r(_record_fire_ownership_lostrV sB"6vj"YVU$?U_`u|ry|r|ry|ry|r|ry|r|dk7ry|r|syy)Nrbqueued suppressednot_configuredlocal deliveredsuppressed_ackedrdelivery_errorrYrZr[r\r*rWrXs r(_classify_delivery_outcomerf s< /+,7g" rf)agent_declaredrargch|r t|nd}t|v}|xst|v}d} d} |rM|sKtjdd|j } ddlm} | |jdxs|d| } n|r|} t|nt||xsd| \} } | rd} n|rSdd lm }||jdxs|d|d|j jd t|z} n7dd lm}t!||t|z|||jd z} | ||| | fS)aMText to deliver for a finished run. Returns ``(deliver_content, blocked_config, silent_alert, incident_acked, failure_incident_id)``; ``silent_alert``: an alert-once marker says the operator was already told, deliver nothing. ``agent_declared``: *error* is the agent's own ``[CRON_FAILURE]`` evidence, delivered verbatim.r+FNz\[blocked_config[^\]]*\]\s*r)blocked_config_noticerFrGr)rr) hold_noticer)rkrrrrr-rrirHrrrrrPrrjr)r<r*rrarrgr9blocked_config_silentblocked_configr\failure_incident_id_pf_textrideliver_contentrrjs r(_compose_run_deliveryrp sL#e*2C8C?*J.Cs.JNNg668"cBHHJE/0L3t9hW (,S1 /K "+/ ++  O  K4,3t9c$i9K9KC9P%c*+O 44S%@CXY\C]]c377+@#ABC  N,A>Sf ffrfceZdZdZy)_FireClaimLostDuringSideEffectzPRaised inside a side-effect fence when the durable fire claim is no longer ours.Nrrrfr(rrrr sZrfrrcTeZdZdZ d dedeedeefdZdefdZ d Z defd Z y) _FireOwnershipzZFire-claim ownership checks for one run (``owner`` is None when the job carries no claim).Nr<rErFc||_||_||_|| t||nd|_|j d}t |tr$t|j dxsd|_ yd|_ y)Nr'r=r+) r<rErFrErrHrJrKrkr)rdr<rErFr5s r(rcz_FireOwnership.__init__ s{% 0%)9)E !-= >  %3=eT3JS4.B/ PT rfrcV|jduxr|jjSr=)rFr?r>s r(transport_cancelledz"_FireOwnership.transport_cancelled- s'$$D0ST5J5J5Q5Q5SSrfc|jtjdSt|jd|jS)NTrGrB)rr! nullcontextrr<r>s r(side_effect_fencez _FireOwnership.side_effect_fence0 s6 :: ))$/ /tzzJJrfc|jry|jyt|jdry|jduxr|jj } t |jd|jry |j|jjy#t$r*tjd|jdd|cYSwxYw)NTFrGrBz0Job '%s': fire_claim ownership validation failedr) rwrr r<rEr?r r#rrr)rdlatcheds r(lostz_FireOwnership.lost5 s  # # % ::  ($ 8//-J$//2H2H2J #DHHTN4::NO ?? & OO   !  LLBDHHTN]a  cN  s#$B000C#"C#)NN) rgrhrirjrKr r;rcrCrwrzr}rrfr(rtrt s\dCG7;UU%-.>%?U"#34U$TTTK drfrtceZdZUdZeed<eed<eeed<dZ eed<dZ eeed<dZ eed <dZ eed <dZ eed <dZeed <dZeed <dZeeed<dZeed<y) _RunDeliveryzUMutable outcome of the save/compose/deliver phase, read back by the bookkeeping tail.r<r*rFdelivery_attemptedNrerYrZrlrgr\rmside_effect_ownership_lost)rgrhrirjrKrrCr rkrrerYrZrlrgr\rmrrrfr(rrN s_ I M C=$$$(NHSM( ND #t# ND !ND  ND )-#-',,rfrdfencerc|j}|j5} | stt|drdnt |d|} ddd|r t j d| |jrt|d|rd|_d|_ t||j|j| |j\} |_ } |_ |_t| j!xr| |_|j"rK|js?|j%dr.dd lm} | |r d|_t j d |d|j"r<|jr0t+| r%t j d |dt,d|_|j"r0|j/r d|_t j1d |d|j"syt3t5||j dk(xrt7||j  |_ |j5}|std|_t=|| |||j |_dddy#1swY*xYw#1swYyxYw#t@$rE}tC|trtE||_t jd|d|Yd}~yd}~wwxYw)zSave output, compose the notice and deliver it (both side effects run under the fire-claim fence; a lost claim raises ``_FireClaimLostDuringSideEffect`` for the caller).rGNzOutput saved to: %sFz`Interrupted by gateway shutdown before the run finished (tool subprocess was killed mid-flight).)r*rrarrgr r) will_retryuAJob '%s': suppressing failure notice — automatic re-run pendingu1Job '%s': agent returned %s — skipping deliveryz;Job '%s': skipping delivery after fire claim ownership lossrXoriginTrUDelivery failed for job %s: %s)#r<rzrrr r rrr*rrrprgrlr\rmrCr-rYrHrrr#rr}r_normalize_deliver_value_delivery_lane_value_resolve_delivery_targetsrZrrYrer#rJrk)rrrarrVrWr9rGr< owns_outputrro _silent_alertr owns_deliverydes r(_save_compose_deliverra s %%C  "5k0 02#d) ?(3t95t<=&* "#)1+;+;.>377KaCb 1 H-7 )* 0 H *3t9 56 D 199agg:6)4:6Ff %I K1'' 67 $SWW-S%T U''--34H^_^g^gZg4hi''  ::199q445aiiqwwIY[ rferr_textctt|d}t||\}}|ryd} t|t ||t |z||d}t| xr|dk(xrt|d } t|d| |d d |jd t|jd  } | d vr t||| fS#t $r/}t|}tjd|d|Yd}~d}~wwxYw)zWFailure notice for a run that raised out of run_job. Returns (delivery_error, outcome).Tr)NrcNrUrrGrFrrrdr)rrrrYrrPr#rkrrrCrrfrHr) r<rrVrWr[r\rmre delivery_excrZrs r(_deliver_crash_failurer s/22FsX\2]^*FsH*U'N''N P(  1h ?BWX[B\ \   A ( * A)#4@ @ 2%dN_-eU 67 $SWW-S%T U W ::23 + ++ P\* 5s4y,OOPs&C C: %C55C:rDrErFrGc V &t||}|j} |j} |j} j d} | s#t ddj dd} d} d}ddlm}m}m }d}d} tdsTtjd j d dt| dd  ||||dd lm}||yyt j"j d| k(}|sAt%| 6tj'dd ||||dd lm}||yy||t)}ddlm}|t)}g&d$&fd }&|| d}|j,|j,|d< t/fi|\}}}}| r4|t3d| |  ||||dd lm}||yyd}|r$j dst5|}|d|d}}}t7|||} t9|||||||||j>|j@}} ||j<r-t3d| |  ||||dd lm}||yy|jBr|jEsd|_!d|_#| r|jI} |j>r(|j@s| stj'ddnH| rtj'ddt3d| |  ||||dd lm}||yytKd|r*tM| | ||||dd lm}||yytO|| | ||||dd lm}||SS#t0$r |wxYw#t:$r d|_YwxYw#|j>|j@}} |wxYw#t0$r}!tQ|!xstS|!jT}"tjGdd|"tS|!|!|!jVfd}#tY|!tZr+| s)tY|!t:s| st]|"||\}}# tKd|s=t_ds/i}$| | |$d<tY|!tZr||$d <tadd|"fi|$n0#tZ$r$}%tjGd!d|%Yd}%~%nd}%~%wwxYw t| d|"|#"n0#tZ$r$}%tjGd#d|%Yd}%~%nd}%~%wwxYwtY|!tZsYd}!~!||||dd lm}||yyd}!~!wwxYw#||||dd lm}||wwxYw)%NrrGr;r<r=Fr)build_profile_secret_scopereset_secret_scopeset_secret_scopeu6Job '%s': one-shot dispatch limit reached — skippingrFz3Dispatch claim rejected; execution was not started.r))reset_terminal_scopeTr@z;Cron job %s lost execution ownership before start; skipping)install_profile_terminal_scopec2D]}t|dy)NrG)r)_deferred_agent_deferred_agentsr<s r(_teardown_deferredz-_run_one_job_body.._teardown_deferredl s$$4 A$_c$i@ Arf)rrrrr)r<r*rrg)rVrWr9rGzWAgent completed but produced empty response (model error, timeout, or misconfiguration)zkJob '%s': fire claim ownership lost after completed delivery; recording the delivered run's terminal statuszbJob '%s': transport cancellation arrived before terminal completion; recording the interrupted runzError processing job %s: %srr_)rVrWrrez/Failed to record interrupted run for job %s: %srz0Failed to finish execution record for job %s: %srB)1rtrrzr}rHragent.secret_scoperrrrrrrtools.terminal_scoperrrIrrr}rrrrmrVrrrrrrrrer*r-rrwrrrrkrVrg __traceback__rJr#rr r )'r<rVrWr9rrErFrGrr_side_effect_fence_fire_claim_ownership_lostrrrerrr _scope_token_terminal_scope_tokenrrOrr _run_kwargsr*rrarrg marker_errorrrwr _err_textrr record_errrs'` @r(rHrH sk 3 ,< =EJ00!&77>*L ' Ih#''BV:WYY]_ NJJL I8 c$i( KKHD * , eK Mp  # | , , A !6 7 -k(FG<W"8"F"N NNXZ]^bZc d`  # | , , A !6 7 -]((BCSCU(VW " ,!??O?Q R"$ A%5((*     )*/*<* ,N, c-c$iI5c$i@ )9CK 56a+4BK 01SYyHKH c LLJCPTIWa b b c d e9O_ a d LLKSQUYXb c c d!Y'   # | , , A !6 7 -q2l  # | , , A !6 7 -s=AO8AO8AO8N(,O8'7O8N=2;O8 B O8O8 O8(N::O8=O OOO!O55O88 VBVA S! V! T*T V TVT"!V" U+U V UV#V VV V(processcdtffd } |jd}|ryt |rytd|d #tj$r|rddlm}||YyYiwxYw) aPreserve ``run_one_job``'s synchronous contract after handoff. The worker owns the durable execution and survives this gateway process. The caller nevertheless waits while it remains alive so manual/background callers do not release their in-process guard or report stale job state. A gateway replacement may kill this waiter; it does not kill the scoped worker or change its ledger ownership. rcbt}t|xr|jdtvS)Nr)rrCrHr)currentrs r( _is_terminalz9_wait_for_external_cron_worker_body.._is_terminal s, -GI H 59I IJJrfTrarr)"reap_terminal_worker_in_backgroundz`cron external worker exited before durable recovery could terminalize its execution state (exit ra)rCr subprocessTimeoutExpiredrrrr)rrr returncoders ` r(#_wait_for_external_cron_worker_bodyr sK$K   c2J > '( > 55?L C  '(( ~]27;   sA (A87A8r)r handoff_filesr.c t|||-t5tjt |ddd|D]} |j dS#1swY$xYw#t $rY0wxYw#|:t5tjt |dddn #1swYnxYw|D]$} |j d#t $rY"wxYwwxYw)NrT missing_ok)rrVr1rr'unlinkr)rrrrrs r(_wait_for_external_cron_workerr3 s2 ,    L,44]65JK L# E   -   L L     L,44]65JK L L L# E   -   sd A3AA$A! $ A0/A03 C<B$ C$B- ) C6C C C CC Cc t|d}t|d}tdz dz }||dz }||dz }||dz }tjdd d t|d t|g}d d lm}m} m} m} d dl m } d dl m } m }d dlm}m}m} t%t'j)dxsij)dd}| }||d|d||}|j,dk(ryt/| t1dt3| |j5dt9j:|t8j<t8j>zt8j@zd} t9jB|dd5}tEjF|ttjI|d||jKt9jL|jOdddtjI}| || ||} || |dd!t|i"}| |||}d#D]}|jU|dd d$l+m,}t[t\jIj^j^}|||} t9j:|t8j<t8j>zt8j`zd} tcjd|jft||tbjhtbjh|dtk%}t9jl| tn5tpjstu|dddtwjxtzz}twjx|kr#|j}r* tEj~|jd} |jSd t| tr| j)d|k7r'tjd(|t|||||f'Stjd)|| j)d*|tn5tjtt5t| j)d*xs |jttu|<ddddddt|||||f'S|j}!|!tn5tpjtu|ddd|jSd d d+lQmR}"|"|}#|jSd |j,d,k(r)||r!t1d-t9jd.t1d/|!d0|#twjd1twjx|kr#tjd2|tzt||||||f'S#t*$rd}YUwxYw#t6$rY wxYw#1swYCxYw#tP$r|jSd wxYw#| |wxYw#t9jl|wxYw#tP$r&|jSd |jSd wxYw#1swY5xYw#t*$r<tjd&|t|||||f'cY|jSd SwxYw#|jSd wxYw#1swYLxYw#1swYQxYw#1swYxYw)3uLaunch *job* outside the managed gateway process when required. Returns ``False`` outside a managed systemd gateway (in-process path). In managed topology the job always goes to an external worker with the #101940 ownership handoff: in a transient user scope, or — when no user D-Bus session exists and ``cron.require_restart_safe_scope`` is false — as a direct subprocess (process separation kept, cgroup isolation lost). rrGr@zexternal-workersz.jsonz.ready.stderrz-mzcron.scheduler--external-worker-file --ack-filer)ris_multiplex_activerrhydrate_profile_secret_sources)build_subprocess_envstrip_launch_profile_env)restart_safe_gateway_child_argvscoped_spawn_lost_user_bussystemd_user_bus_envrequire_restart_safe_scopeFzcron-z-exec-) unit_suffixr in_processNz;cron execution claim changed before external worker handoffiwrr)r<rmultiplex_activeTr HERMES_HOME) scrub_secretsinherit_profile_homeextra)HERMES_INTERACTIVEHERMES_GATEWAY_SESSIONHERMES_EXEC_ASK)pin_hermes_tree_on_pythonpath)renvstdinstdoutstderrstart_new_session creationflagszhCron external worker %s published an unreadable acknowledgement; treating handoff as ownership-uncertain)rrrz]Cron external worker acknowledgement mismatch for %s; treating handoff as ownership-uncertainz?Cron job '%s' handed to restart-safe worker pid=%s execution=%spid)external_worker_stderr_tailscopedzUrestart-safe systemd scope could not be created: the user D-Bus session at /run/user/z/bus disappeared after the gateway started. On a system-level service install, run `sudo loginctl enable-linger `; the next fire dispatches without scope isolation.zCcron external worker exited before ownership acknowledgement (exit rag?zqCron external worker for job '%s' did not acknowledge within %.0fs; leaving the durable execution claim untouched)Vrkr}sys executablerrrrrrrtools.environments.localrrtools.process_registryrrrrCrrHr#rrrrchmodrrrO_WRONLYO_CREATO_EXCLfdopenrdumprLflushfsyncfilenormrrcron.scheduler_worker_envrr__file__rO_TRUNCrPopenargvDEVNULLrr#rVr1r}r'rrrr_loads read_textrrrrJrKrrr!r"rr/rIrr2pollrrrgetuidsleepr)$r<rr handoff_dir payload_pathack_path stderr_pathrrrrrrrrrrrrrdispatchfd payload_filer secret_token worker_env _presence_varr repo_root stderr_fdrdeadlineacknowledgementrr stderr_tails$ r(rJrJK ss>*+L T^F"$v-0BBKL>!77L ~V44H<. 88K    L H G EW +%) ! # ' ' / 52 : :;WY^ _& " +,.F86,8#=H }} $%l3; I  [! %  r{{RZZ7"))CU KB YYr3 1 ,\ II$'(8(:(B(B(D$E(8       HH\((* + ,$%--/L"<0#$>|$LML)-.B*!% #l"34/  <(%j1J , }d+ ,HX&&(//66I.z9EJGGKrzz)ABJJ)NPUV   &&  N ((!)) "&02 G HHY  @$((v)>?@~~"@@H .. X % ??  1"&**X-?-?-?-Q"R405"&&~6,F >  6!-!#/"=  KKQ##E*    ? 3 3Iz J ?>A#''.='++??$]6%:; ? ?2)+[9   \\^  ! L,44]65JK L   4  0 N5kBK   $  /}}(-G -S#!!# .HH # Ak]4  4K .. X %V NN 8&  *!#X{;  ] +%*"+&     , , t,  <(> HHY  t,d+  @@   >  6!-!#/"= 40 40, ? ? ? ? L Ls6XX",X?A-X21X?(Y.AZ6AY, Z&Z8%[\1!9\$\1\> XX" X/.X/2X<7X??Y Y),ZZ/Z58[0\ 5\ \  \ \!$\. )\11\;>]rrc  tj|jd}|d}t|dj }t |d} |jdd d l m }m }m}m} m} d d lm} d d lm} d dlm} m}||}|}t1|j3dd }| || || ||} t5|5| |8tj7d| ddd||| || |y |j8j;dd|j=|j>dtAjB}tAjD|t@jFt@jHzt@jJzd} tAjL|dd5}tjNtAjB|d||jQtAjR|jUdddtAjV|| t@jZj3d}|t@jZd< t]|ddd |!t@jZj_ddn|t@jZd<tajbt5|jedjddddcddd||| || |S#t $r;tjd|Y |jdy #t$rYy wxYwwxYw#t$rYwxYw# |jdw#t$rYwwxYwxYw#1swY{xYw#tX$r|jdwxYw#t $r9tjd|Yddd||| || |y wxYw#1swYxYw#|!t@jZj_ddn|t@jZd<tajbt5|jedjddddw#1swYwxYwxYw#1swYnxYw ||| || |y#||| || |wxYw)a Adopt and execute one gateway-dispatched cron payload. The execution row is created by the gateway before spawn, then transferred here before the ready acknowledgement is published. No side effect runs unless that durable ownership transfer succeeds. rrr<rrz.Cron external worker could not load payload %sTrFr)rrrset_multiplex_activer)adopt_claimed_executionr)reset_hermes_home_overrideset_hermes_home_overriderNzUCron external worker refused execution %s: durable ownership could not be established)parentsexist_okz.tmprr)rrzCCron external worker could not publish ready acknowledgement for %sr@rVrWr9r)3rrrrrLrkr#rrrrrrrrrrrrrrrrrrCrHrrrmkdir with_namerFrgetpidrrrrrrrrrreplacermrIrSrr!r" with_suffix)rrpayloadr<rrrrrrrrrrr home_tokenprevious_multiplexrr  ack_tmp_pathr ack_fileold_external_executions r(_run_external_worker_payloadr(-s **\33W3EFenGN34<<> 3~./     4  08D *,7J,.GKK(:EBC)*"<0#$>|$LML3/ L ). L&|4< /  . L` <(/0":.U %%dT%B (11X]]O4 }2UV WW\2;;+Cbii+OQVW2sW=4 "))+|"TV^_ (!234JJ|X6&(ZZ^^4R%S "9EBJJ5 6 L"3D%P)1JJNN#A4HAWBJJ=> ((1L((3::d:KL[. L. L` <(/0":.a I<X    4  0            4  0   V44 % ''4'8   Y A. L` <(/0":./ $LL*1JJNN#A4HAWBJJ=> ((1L((3::d:KLLL[. L. L. L` <(/0":. <(/0":.s6AL+M2 S R2S"BO9N6AN)0N63R2P(AR2"PR2 S+M/ N M M+*M+.M//N2 M?>M?N&NN& N# N&"N##N&)N3 .N66OOP5R26SPR2P% R2(AR/8"R#  R/#R, (R//R22R;7SS2c ddlm}|jy#t$r }tj d|Yd}~yd}~wwxYw)aBest-effort: tell the active scheduler provider the job set changed. Call AFTER a successful store mutation so an external provider can re-provision/cancel the one-shot; no-op for the built-in. Kept out of cron/jobs.py (import cycle). Never raises.rresolve_cron_schedulerz!on_jobs_changed notify failed: %sN)cron.scheduler_providerr+on_jobs_changedr#rr)r+rs r(_notify_provider_jobs_changedr.s;=B 002 = 8!<<=s AAAcHeZdZdZdededdffd ZdefdZdefdZ xZ S) CronSchedulerRegistrationErrorzFA job was persisted but its first external trigger was not registered.r<causerNcz||_||_t| d|ddt |j dy)NrrGz:' was saved, but its first scheduler registration failed (zS). Do not create a duplicate. Pause/resume or update the job to retry registration.)r<r1rbrcrVrg)rdr<r1res r(rcz'CronSchedulerRegistrationError.__init__sJ  T $$$(K$8$8#9:O O rfcf|jjdxs|jd}d|dS)zEHuman-facing variant for chat/CLI surfaces (no exception class name).rFrGzSaved cron job 'u', but couldn't register it with the external scheduler yet. The job is kept — don't re-create it; pause/resume or edit it (e.g. via /cron) to retry registration.)r<rH)rdrjs r( user_messagez+CronSchedulerRegistrationError.user_messages9 V$6ug&N N rfc>t||jdddddS)zDReturn the public partial-failure contract without provider details.rGTF)rr job_savedscheduler_registered retry_create)rkr<r>s r(to_dictz&CronSchedulerRegistrationError.to_dicts)Yhhtn$)! # #rf) rgrhrirjrKr#rcrkr4r9rlrms@r(r0r0s8P D  t  c ##rfr0c ddlm}ddlm}|di|}|j dds|S |j ||S#t $r}t|||d}~wwxYw)zHPersist one job and register its first trigger with the active provider.r) create_jobr*enabledTNr)rr;r,r+rH register_jobr#r0)rr;r+r<rs r(&create_job_with_scheduler_registrationr>si$>  v C 779d # @ --c2 J @,S#6C?@sA A AAgr@_last_dead_owner_reap_atg@_last_worktree_maintenance_atc t}tjt5t t j jj}|dz jr|jt|dddtjt5ddl m }|D]}t|jdxsdj}|rt |jsK t!j"gdddd d d | }|j$dk(rC|j&jr)|j|j&j dddt)|Dcgc]!}t |d z js |#c}S#1swY.xYw#t$rYwxYw#1swY\xYwcc}w)zRepos whose ``.worktrees/`` to keep pruned: the hermes checkout plus job workdir repo roots, filtered to those that actually have a ``.worktrees/`` dir.z.gitNrrarr+)gitz rev-parsez--show-toplevelTrr )capture_outputrrerrorsrgrz .worktrees)rr!r"r#rrrLrr_r}rkrrbrHr-rrr!rrr)repos install_rootrbr<rprobers r(_worktree_maintenance_reposrIsE   Y ')H~--/66== 6 ! ) ) + IIc,' ()   Y ''; C#''),2399;G$w-"6"6"8 ";#'dW$aW>##q(U\\-?-?-AIIell0023 "e} J!a<)?(G(G(IA JJ-))&  " KsJAF/$AG >A0F<.G !G(G/F9< G G G  G  Gctj}t5t|tz tkr dddy|addddd}t j |ddjy#1swY4xYw)zThrottled worktree prune from the cron tick, on a daemon thread so the tick never waits on git. Same conservative pruner as ``hermes -w`` startup (dirty/unpushed/locked trees untouched). Errors never propagate: GC is hygiene, not scheduling.Nc t}|syddlm}|D] } || y#t$rtj d|dY2wxYw#t$rtj ddYywxYw)Nr)_prune_stale_worktreesz'Cron worktree maintenance failed for %sTrz!Cron worktree maintenance skipped)rIclirLr#rr)rFrLrepos r(_runz-_maybe_run_worktree_maintenance.._runs M/1E 2 aa*40 a!aLL!JD[_L`a M LL ? NN7>>+V__a @   $$W-     (- OP S ! LL@   LLDc J s<A B :B D7D2.C? D2C 'D299D22D7ctrMtjttf5tj |tj dddnatr[tjttf5tj|jtjdddd|jy#1swYxYw#1swY&xYw)NrE) r<r!r"rIOErrorrTLOCK_UNr@rWrLK_UNLCKr#)rZs r(_release_tick_lockr`9s  '7!3 4 0 KK / 0 0   '7!3 4 A NN7>>+V__a @ A MMO  0 0 A As%C94CCCcRtt}tj}tj |}| ||z t kry|t|< ddlm}|}|rtjd|yy#t$r }tjd|Yd}~yd}~wwxYw)aDead-owner reclaim: a run that died mid-flight would leave its row 'claimed' forever. Rows whose owner process is proved gone are released (_owner_is_live), as are rows whose live owner holds a claim older than the derived stale bound (the process is not killed). Throttled.Nr)rziReclaimed %d cron execution(s) whose owner process died before reaching a terminal state (marked unknown)z'Dead-owner execution reclaim failed: %s) rr}rrr?rH!_DEAD_OWNER_REAP_INTERVAL_SECONDSrrrrr#r) _reap_key _reap_now _last_reapr _reclaimed _reap_excs r(_maybe_reap_dead_ownersrhCs 0 23I I)--i8J  "%F F*3Y' KB35  NND   K > JJKs%A== B&B!!B&ctttfdtDsy|}t j t 5tDchc]}|dk(s |d}}|Dchc]$}t|ts|jd&}}||ks ddl m }|}ddd t|ycc}wcc}w#1swY xYw#t $r }tjd|Yd}~yd}~wwxYw)zBound the in-flight set BEFORE the dedup guard so a leaked claim is force-released now rather than eating every later fire until restart. Skipped when nothing is in flight.c3.K|] }|dk(yw)rNr)rLrTrs r(rMz1_sweep_stale_inflight_for_tick..gsAs1v$AsNrrErGraz Stale in-flight sweep failed: %s)rr}rNr/r!r"r#rJrKrHrrbrrr) r _sweep_jobsrT _inflight_idsrp_due_ids_load_all_jobsrrs @r(_sweep_stale_inflight_for_tickrocs""2"45K A0@A AK   Y '++;UCs1v?TQU U)1IAZ45HAEE$KII( =(*K +>[)VI++ >91==>sN C  B? B?'C -CCC 3 C? C  C C>C99C>c tdj}|rt|xsdS tjt5txsi}t|tr|jdinijd}|t|xsdcdddS dddy#ttf$rt j dYwxYw#1swYyxYw)zXMax workers: env > config.yaml > unbounded (HERMES_CRON_MAX_PARALLEL=1 restores serial).HERMES_CRON_MAX_PARALLELNz?Invalid HERMES_CRON_MAX_PARALLEL value; defaulting to unboundedr@max_parallel_jobs)rr-rIr/rrrr!r"r#rrJrKrH)_env_parr_cfg_pars r(_resolve_max_parallel_workersruwsZ#$>?EEG x=(D (    Y ') #-7t-DEIIfb)"QQRef  x=(D )) )   "ZXYZ) s)B,AC,$CCCcx ddlm}|y#t$r }tj d|Yd}~yd}~wwxYw)zReap MCP stdio orphans (only PIDs flagged by tools.mcp_tool._run_stdio's finally block); run AFTER jobs finish so live sessions are never touched.r)_kill_orphaned_mcp_childrenz'Post-tick MCP orphan cleanup failed: %sN)tools.mcp_tool_lifecyclerwr#rr)rw_es r(_sweep_mcp_orphansrzs5DH#% D >CCDs  949ct|dd}|st|dddyt|tr t|n t|}|d|d<|j d|d<t |||| S) z4Run one due job via the shared ``run_one_job`` body.rGT) return_jobrFz+Fire claim lost; execution was not started.r)r<r)rrrJrKrHrS)r<rVrWr9claimed claimed_jobs r(_process_due_jobrs}!Ttr?r;Nz\Could not clear run_claim for job '%s' after dispatch failure: %s (claim will expire at TTL))rHrJrKrr#rr) _schedule claim_errr<r job_labels r(_clear_run_claim_best_effortz8_submit_with_guard.._clear_run_claim_best_effortshGGJ' 9d+ f0E0O  & F # & NN99 & & &s A A/A**A/c2tjdy)Nu8Job '%s' not dispatched — interpreter is shutting downr3)rsr(_not_dispatched_shutdownz4_submit_with_guard.._not_dispatched_shutdownsQS\]rfu%Job '%s' already running — skippingbuiltinr<r=rr*z6Job '%s' not dispatched: execution creation failed: %sct |j|t|d|S#t|d|wxYw)NrGr*)r!r)rpctxr process_jobs r(_run_and_releasez,_submit_with_guard.._run_and_releases4 477;* $d 3 $d 3s%7FzExecutor dispatch failed: r)zJob '%s' not dispatched: %srB)rHr2rrrr}rrKrrr#rrr rrJrrrVr'r/r5)r<rrrr _claim_homerNdispatched_job_ctx execution_errrr submit_err _submit_keyrrs` ` @@r(_submit_with_guardrsYF'I&0^"# "$& #F + ;YG#$K $ 9@T8UW c $@'')*t+4 kk*+ 0#FK8 * *,/ [ )0 J? F5$& DiQ^ `   F5$& dOU4Nzl2[ ] j, /4Nz4Z $ & LL6 : N 0 Js>0AC:;D;F3: D8+D33D8; F0A"F++F03F=rc|s tyt|gdtjjddffd }|D]}|j |y)zAsync (gateway ticker) mode: sweep via a done-callback after the LAST job completes; sweep inline when nothing was dispatched (all skipped / no due jobs).N_frc"dxxdzcc<tjt5|j}|.tj d|t |||jfdddddkr tyy#1swYxYw)NrrEz(Cron job future failed in async mode: %sr) r!r"r#rrrrVrrz)r_exc _remainings r(_on_donez2_sweep_mcp_orphans_when_all_done.._on_dones1    + E<<>D >"4j$0B0BCE E a=A      E Es ABB)rzrrrFuturer")rrrrs @r( _sweep_mcp_orphans_when_all_donersZ g,J !Z''.. !4 !' X&'rf)tick)rYrrrr)rrr4)rrrr)rrrrrr__main__r)add_helpT)rVrequiredr) setup_loggingr@)r6rrE)r9)cron.scheduler_deliveryBOT_CHAT_PLATFORM)zcron.scheduler_preflightSharedRouteAdapters)rcron_delivery_targets)rparse_bot_chat_deliver_token)rrrrctj|}|tdtd|ddl}ddlm}|t|g|t|j|d|dS)Nzmodule z has no attribute r) warn_oncerE) _PLUGIN_COMPAT_LAZYrHAttributeErrorrg importlibhermes_cli.plugin_compatrr0 import_module)rFrqrrs r(rrUse $ $T *F ~wxl2DTHMNN2 h&v& 9**6!95vay AArf)r&zconcurrent.futures.FuturerNr=)rzfrozenset[str]rB(8rjatexitconcurrent.futuresrr!rr;rloggingrrrrrtrr dataclassesrrrrr< ImportErrorr@pathlibrtypingrr r r r r rrinsertrkrrrrrcron.env_settingsrhermes_cli._subprocess_compatrrdrrrr hermes_timerragent.interrupt_compatragent.delegation_contextrragent.memory_providerragent.turn_failure_copyr getLoggerrgrr)r7r;rKrPtuplerUrZrr\compilerrrvrrwrKrrrrrrrrCrrrrr#rrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrr#r$rrr%rIr'r(r+r.rr/r0r1r2LockrVr3r4r5r~r6r7rrr9r;rErYr^rurxrrrjrrrkrrrrrrrrrrurr rrrregisterr r(r.rmr2r3r}r8rrBrFrJrPr`_DEFAULT_SCRIPT_TIMEOUT_SCRIPT_TIMEOUTrDr.r-rhryr{rrrrrrrrrr rrr'r6r^rvrrrrrrrrrrrrrr8rSrUrVrfrprrrtrrrrrrHrrrrJr(r.r0r>rbr?rQr@rPrIrRr[r`rhrorurzrrrcron.scheduler_tickrcron.scheduler_deliveryrYrrrrcron.scheduler_scriptrrr4cron.scheduler_promptrrrrcron.scheduler_preflightrrrrrrrargparseArgumentParserparser add_argument parse_argsrhermes_loggingr SystemExitexternal_worker_filer&asyncioshutilsignalrrrrfr(rs    !22GGG3tH~,,3345=.<'9*9K+<   8 $ !B$t:5c?T#93: l &"**OP ?C$J ?5c?T;Q ?%%'c6k"'#U38_t-C#L 4 4 4BdB3:B#BJ&)"hUh9T9d9(<@ +3C= 4# 0hdhth Q Q$ Q  $s)0d3idtCy$4DI6VtV$VsVtd{V.aaaa$$$  ' 6S 6Xc] 6 0C 0D 0$EGc:--@@@AF79Dhsm!349h#hXeD#I.>%?h5h)+d39o*$4 @#@$@#NPd5$vuXc]D5H/I'I"JJKP%(Ec)*,d5#:&+  d$(s$#' Ec!x   [  d  $[N F3 FhuT3Y/?&@ FD FST6,,8E$)4D+E,QU,"++4 d%,t& >$ >F3FcFFZ_FdhF($4 t E d  Tt TS Tu T T]` Tei TF?!8D>?!T?!F267 7!)#7 7t+C+(8+D+ c (62B d !"e),7<FKS\SbSb"d(# 0 % HSMj6H6H6[6[& <''(=4=RR :t : :*H]$;*t*$! dTk -$--tTz*- 7 t CCC ,=,T,&"C()#&BQ&F##& u&MQ''$'':B5/' 'T&&Rc3CCdCHSM4> 4>4>&)4> 4c8C= ()4>n#2 #2#2&)#29A##2 8E?HSM8C= 89#2L  2Kt2KS2KC2KN2Kjchtn0@T@3@#@D@XV[_@><3<#<3<5<6AXdAXCAX^AXdTWiHXAXH4"```"+t+\,s,U,t,.$(kk!k+.k:=kHKk4.k k\;;c;S;VY;|VVcVVSVV[^VVcgVVrc34c8C=0 1 U UU&)U9A#U 8J # ./Up* C(C(L D T &  * *  *%4%%%O%P""D""J2:>]a/3SWy4 y4(0y4MUVY]y4+,y4CKC=y4 4c8C= ()y4z?C)),4UO) )8Q+Q+dQ+j dE"&SWG> G>59G>3-G>6>?O6PG> G>TT L L# LVY L^b L %'+@D-1<@48 0!.g .g.g8;.g.g 3dD(3- /0.gb[Y[33l -- -$PFPF*PF  ,%*!-/$sEz*/*4&15x5+Y^^-KT#YK>SB!HK@>T>d>( x}  D S$ S S$ SYDY (:(:(M(MYx'd't'.%$ z388+(((%844$OLtdC  "   4 4 &6&8v F-d.G.GWA  ]^    JNQ$_ ByA E  FA   sH h/1i/i 7h<;i <ii ii  i ii