+ jHj'2a0t$Rt^RIHt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI H t H t ^RIHt^RIHt^RIHtHtHt]P*!R4tRt]!]P2]P4]P6]P8]P:04tR;tR t R R lt!R R lt"] !RR44t#]!RR04t$/t%R]&R&/t'R]&R&] PP!4t)Rs*R]&R&RRlt+RRlt,RRlt-RRlt.RR lt/RaDegraded state for an unwritable cron store (ENOSPC, EROFS, EACCES). A full disk or read-only mount fails every store write on every 60s tick. Instead of one warning per failing call site, each store directory gets ONE in-memory record: set by the first failed write, updated by later ones, cleared by the first jobs.json save that lands. While it is set, the tick skips the advance/claim work that can only fail and re-probes the store at most once a minute. ``probe_store`` is also what ``hermes cron status`` and ``hermes doctor`` use: they run in another process, so they cannot read this record or trust markers in a directory nobody can write. ) annotationsN) dataclassfield)datetime)Path)CallableIterableOptionalz cron.jobsgN@zEfree disk space, remount it read-write, or fix permissions on {store}c V^8dQhRRRR/#)excOSErrorreturnstr)formats"6/home/ubuntu/.hermes/hermes-agent/cron/store_health.py __annotate__r#s///C/c\PPVP;'g^R4pVP;'g \ V4pV'dV RV 2#T#)z: )errno errorcodegetstrerrorr)r codetexts& rdescribe_errorr#sO ??  syy~~Ar 2D << # #3s8D $dV2dV .$.rc(V^8dQhRRRRRRRR/#)r storererrorsinceOptional[float]rdictr)rs"rrr)s.ff#fcf/fdfrc VfRM3\P!V4P4PRR7pRVRVRVR\P VR7/#) z\The fields `cron status`, doctor and the home-channel notices print for an unwritable store.unknownseconds)timespecr r!r"fixr )r fromtimestamp astimezone isoformatFIX_HINTr)r r!r" since_texts&&& r_report_fieldsr0)sS#m1G1G1N1Y1Y1[1e1eox1e1yJ UGUGZ^cHd eerc]tRt^/t$R]R&R]R&R]R&RtR]R&]!]R 7tR ]R &]!]R 7t R ]R &Rt R]R &] RRl4t RRlt RtR#) StoreDegradedrr floatr"r!Nr# recovered_at)default_factorysetsitesskipped last_probecV^8dQhRR/#)r rintr)rs"rrStoreDegraded.__annotate__<s!!c!rc ,\VP4#N)lenr8selfs&r skipped_runsStoreDegraded.skipped_runs;s4<<  rcV^8dQhRR/#)r rr$r)rs"rrr<?scctcrc \\VPVPVP4VP R7#))r8)r$r0r r!r"rBr@s&r notice_fieldsStoreDegraded.notice_fields?s+N4::tzz4::FPTPaPabbrr)__name__ __module__ __qualname____firstlineno____annotations__r4rr6r7r8r9propertyrBrF__static_attributes__rrrr2r2/sc J L J$(L/(s+E3+-GS-#'J& !!ccrr2advanceclaimzdict[str, StoreDegraded] _degraded _recovered.Optional[Callable[[str, StoreDegraded], None]] _listenerc V^8dQhRRRR/#)r fnrSrNoner)rs"rrrNs NSWrc VsR#r>)rT)rVs&rset_transition_listenerrYNsIrc$V^8dQhRRRRRR/#)r eventrrecordr2rrWr)rs"rrrSs&WW3W W$Wrc\pVfR#V!W4R# \d\PRTRR7R#i;i)Nz(Cron store %s transition listener failedT)exc_info)rT Exceptionloggerdebug)r[r\listeners&& r_notifyrcSsCHW W ?QU VWs #??cV^8dQhRR/#)r rrr)rs"rrr]s**$*rc0^RIHpV!4P#)r_current_cron_store) cron.jobsrgcron_dirrfs r_active_cron_dirrj]s-  ) ))rc V^8dQhRRRR/#)r riOptional[Path]rrr)rs"rrrbsVV>VcVrcd\PPVeV4#\44#)zRecord key for a store: its resolved path (default: the active store). Callers pass both resolved (cron store) and unresolved (tick-lock dir, profile homes) paths for one store.)ospathrealpathrjris&r_keyrrbs* 77  (\P\V44#r>)rQrrrrqs&rdegraded_recordrs ==h ((rc(V^8dQhRRRRRRRR/#)r rirdue_atr3gracerboolr)rs"rrrs.YYDY%YY$Yrc faa\;QJdZVV3Rl\P\V44\P\V4434F 'gK R# R#!VV3Rl\P\V44\P\V44344#)zVWhether a run due at ``due_at`` fell inside this store's current or just-ended outage.c3<"TFTpVRJ;'dDVPSS,8*;'d&VPRJ;'gSVP8*xKV R#5ir>)r"r4).0rrrs& r outage_covers..s\YW1}ssFUN!:ssRV@V@r@rZ`dedrdrZrsWsAAAATF)anyrQrrrrR)rirrs&ffr outage_coversrs 3Y"tH~6 tH~8VWY33Y3Y3Y"tH~6 tH~8VWY YYrcV^8dQhRR/#)r rrr)rs"rrrs%%s%rc8\\V4R, 4#)zQRecord key of a profile home's cron store (a symlinked cron/ resolves elsewhere).cron)rrr)homes&r store_keyrs T V# $$rcV^8dQhRR/#r rrWr)rs"rrrs##T#rc\V4p\;_uu_4\\3F&pVUu.uF q"V9gK VNK upFpWK K( RRR4R#uupi +'giR#;i)zDrop the state of ``stores`` (``store_key`` values, resolved while each home existed) whose profile this process no longer ticks, so a profile that left this gateway cannot keep the host-wide gauges at writable=0.N)r6rrQrR)storesrecordssr s& r forget_storesrsY[F !:.G%,)rrrQvaluesrrrdegraded_recordsrs$ I$$&' s 7 A c V^8dQhRRRR/#)r due_jobsrrrr)rs"rrrstrc\4p\P\V44pVeVPfR#\ P !4pW2P, \8pV'dRM \V4p\;_uu_4V'gW2nVe\V4Vn T;'gVRJpV'd#V;P\V4,un RRR4V# +'giX#;i)a Whether this tick skips advance/claim for ``due_jobs``: the active store is known unwritable and the minute-throttled re-probe has not seen it accept a write. Skipped runs are recorded; once the probe passes, the dispatch's own save confirms recovery (or re-degrades).NF)rjrQrrrr9rrPROBE_INTERVAL_SECONDS probe_storerrr!r8rz)rrir\now throttledr!blockeds& rdispatch_blockedrs !H ]]4> *F ~**2 .. C'''*@@ID;x#8E  #   )%0FL00uD0  NNi1 1N  N  Ns+C3: C3"C33 D c V^8dQhRRRR/#rr)rs"rrrs11$141rc\'d9\;_uu_4\P\V4R4RRR4R#R# +'giR#;i)zLA due scan's save landed after the outage ended: normal grace applies again.N)rRrrrrrqs&rend_recovery_windowrs2z U NN4>4 0U UUs A A cV^8dQhRR/#rr)rs"rrrs--d-rcZ\'gR#\4p\P\V44p\P !4pVe*VP eW!P , \8dR#W!n\V4pVf3^RI H pH pH pV!4;_uu_4V!V!44RRR4R#\;_uu_4\V4VnRRR4R# +'giR#;i \d pTpRp?LORp?ii;i +'giR#;i)aGIdle tick (nothing due, so no save will land): re-probe a degraded store at most once a minute. The probe only creates an empty file, so a passing probe is confirmed by a real jobs.json save, whose ``note_writable`` ends the outage; metrics, notices and the one-shot grace gate then stop treating a recovered store as degraded.N) _jobs_lock load_jobs save_jobs)rQrjrrrrrr9rrrhrrrr rrr!)rir\rr!rrrr s r recheck_idlers 9!H ]]4> *F .. C ~&++7CBSBS> )+&  %e,     E  sBC?*C+8C?D+ C< 6C?<C?? D DD D* c V^8dQhRRRR/#)r rirrzOptional[dict]r)rs"rrrs G G4 GN Grc\V4pVfR#VR, P4Pp\ \ V4\ V4V4# \dRpL.i;i)zCross-process view for `cron status` / `doctor`: ``None`` when the store accepts writes, else the notice fields. ``since`` is jobs.json's mtime (the last write that landed); ``skipped`` is left to the caller, which counts the due jobs that have not fired.N jobs.json)rstatst_mtimer r0rr)rir!r"s& r probe_reportrsd  !E }K'--/88 #h-)> FF s!A A#"A#c V^8dQhRRRR/#)r rorrz Optional[int]r)rs"rrrs  T m rc.\\R4'dN\P!4^8Xd4\P!V4pVPVP ,#\ P!V4P# \dR#i;i)zBytes this process can still write. Root may also fill the reserved blocks that ``disk_usage().free`` leaves out, so a 'full' disk is not full for root.geteuidN) hasattrrnrstatvfsf_bfreef_frsizeshutil disk_usagefreer )rostatss& r free_bytesrsk 2y ! !bjjla&7JJt$E==5>>1 1  &+++ sA"B%B BBc V^8dQhRRRR/#)r rirrzOptional[OSError]r)rs"rrr s!!$!#4!rc (VP4'gR#^RIHpVR, pVP4PR,p\ PPV4\VR, 43Fp\ PPV4'gK*\ P!V\ P4'dKW\ \P\ P!\P4V4u# RpRR,pV!VRR 7wru\ P !VR 4;_uu_4p\#^V\%V44Fp VP'VRW9, 4K VP)4\ P*!VP-44RRR4TeD\.P0!\ 4;_uu_4\ P2!T4RRR4R#R# \ dRpELi;i +'giLn;i \ dfp T uRp ? TeV\.P0!\ 4;_uu_4\ P2!T4RRR4# +'gi#;i#Rp ? ii;i +'giR#;i TeV\.P0!\ 4;_uu_4\ P2!T4RRR4i +'gii;ii;i) aUWrite probe mirroring a jobs.json save: ``None`` when it would land (or the store does not exist yet), else the OSError it would hit. Stages a payload the size of the current jobs.json (plus a page) where the save stages it (beside a symlinked file's real target), so a full, over-quota or read-only target fails here the way the save would.N)mkstemp_besideriz .tick.lockiz.probe_)prefixwb)is_dirutilsrrst_sizer rnrorprexistsaccessW_OKrEACCESrfdopenranger?writeflushfsyncfileno contextlibsuppressunlink) rir jobs_filesizetargettmpchunkfdfoffsetr s & rrr s ??  $;&I~~''$. 77##I.H|4K0LM 77>>& ! !"))FBGG*D*D5<<U\\)BFK KN C EME  9= YYr4 A4U4nt}-.5 GGI HHQXXZ ! ?$$W-- #.   - !  ?$$W-- #.--- .-  ?$$W-- #.--- s!H+H/ A,H6H/$J" HH H, 'H/+J6,H// J:J;J<J6&J J JJ6" J3 6'LK= 4 L= L  Li@)rNr>)=__conditional_annotations____doc__ __future__rrrloggingrnr threadingr dataclassesrrrpathlibrtypingrrr getLoggerr`r frozensetEROFSrEPERMENOSPCEDQUOTUNWRITABLE_ERRNOSLOW_FREE_BYTESr.rr0r2rrQrLrRLockrrTrYrcrjrrrzrrrrrrrrrrrrr)rs@rrsH#   (//   ; 'u{{ELL%++u||UZUaUabc R/ f  cc c&Y01&( #((* $)<@ 9@ W* V EX> ! )Y % #( .1-4 G !r