+ jHjjna0t$Rt^RIHt^RIHt^RIt^RIt^RIt^RI t ^RI t ^RI t ^RI t ^RI t ^RIHt^RIHt^RIHt^RIHt^RIHt]P0!]4tRNtROtR tRNtR tRPt RQt!R t"R t#^t$R t%RRt&Rt'Rs(R])R&] PT!R4t+RRlt,]-!0RSm4t.RRlt/RRlt0Rt1RRlt2Rs3R])R&RRlt4RR lt5Rs6R!])R"&R#t7]73R$R%llt8R&R'lt9R(t:R)R*lt;R+R,ltR3R4lt?R5R6R7RR8R9/R:R;llt@RTR<R=lltAR>R?ltBRUR@RAlltCRURBRClltDRDREltERFRGltFRHR0/RIRJlltGRK^/RLRMlltHR#)VuLive hardware budget probe. Budget-source rule: discrete cards may trust the device query (measured honest within rounding); unified-memory devices must budget from OS free physical memory minus headroom — their device queries have been observed off by 3x in both directions. Every probe here must work under a stripped PATH — gateway and service sessions don't inherit the interactive environment. ) annotations)suppressN)replace)PathGGML_DEVICE_GPU)HardwareBudget) gpu_classg ףp= ?g?g?g?igN@z3tuple[float, tuple[int, bool | None] | None] | None_pool_probe_cachez/CUDA\d+:.*\((\d+)\s*MiB,\s*\d+\s*MiB free\)\s*$c V^8dQhRRRR/#)argvstrreturn)formats"F/home/ubuntu/.hermes/hermes-agent/hermes_cli/local_runtime/hardware.py __annotate__rIs  3 3 c b\P!\V4RRRR^RR7P#)Tutf-8rF)capture_outputtextencodingerrorstimeoutcheck) subprocessrunliststdout)r s*r_stdoutr!Is/ >> T 4dWY`a  f rcV^8dQhRR/#r rz str | Noner)rs"rrrSsZrc^\R4PRR7# \dR#i;i)z6Raw /proc/meminfo, or None when procfs cannot be read.z /proc/meminfor)rN)r read_textOSErrorrrr_linux_meminfo_textr'Ss2O$...@@ s  ,,c V^8dQhRRRR/#)r rrrztuple[int, int] | Noner)rs"rrr[s  # *@ rc /pVP4FwpVPR4wr4pV'd V\9dK,VP4p\ V4^8wgV^,R8wdK\\ V^,4R,W&Ky VP R4pRV9d VR,pMVP R4pVeV^8:gVe^Tu;8:d V8:gR#R#Wx3# \\\3dR#i;i)u1(total, available) from meminfo text, or None if it cannot be trusted. MemAvailable includes reclaimable page cache, which ``getconf _AVPHYS_PAGES`` counts as used. Kernels that omit MemAvailable fall back to MemFree. Zero is a real available value — a missing-field check must not treat it as absent. :kBiMemTotal MemAvailableMemFreeN) splitlines partition_LINUX_RAM_KEYSsplitlenintgetAttributeError TypeError ValueError) rfieldslinename separatorvaluepartstotal availables & r_linux_ram_from_meminforA[s FOO%D%)^^C%8 "DUO ;KKME5zQ%(d"2uQx=4/FL& :& V #~.I 9-I Mz  *U*+ Iz 2sCC)#C)&C))DDc$a^RIo!V3RlRSP4pV!4pSPV4VnSPP P SPV44'gR#V# \\3dR#i;i)z*GlobalMemoryStatusEx, or None off Windows.Nc<]tRt^tRZP3RZP3.RUu.uFqSP 3NK up,tRtR#uupi)._windows_memory_status..MEMORYSTATUSEXdwLength dwMemoryLoadrN) ullTotalPhys ullAvailPhysullTotalPageFileullAvailPageFileullTotalVirtualullAvailVirtualullAvailExtendedVirtual)__name__ __module__ __qualname____firstlineno__c_ulong c_ulonglong_fields___static_attributes__)r;ctypess0rMEMORYSTATUSEXrDsW &..1NFNN3ST?YZ?Ydf001?YZZZsA rW) rV StructuresizeofrEwindllkernel32GlobalMemoryStatusExbyrefr6r&)rWstatrVs @r_windows_memory_statusr_~sz[))[  n5 }}%%::6<<;MNN G $sAA:8A::BBcV^8dQhRR/#)r rztuple[int, int]r)rs"rrrs''O'rcvaa \4pVeVPVP3#\PR8Xd\ \ RRR4P4;'g^4pV^8:dR#V^,p\\\4;_uu_4\ R4o \P!RS 4pV'd\ VP^44MRp\VV 3R lR44pV^8d WT,pRRR4W3#\PPR 4'd \!4pVe\#V4pVeV#\ \ R R 4;'gR 4p\ \ R R4;'g^4V,pV^,p\\\4;_uu_4\ \ R R4;'g^4V,;'gTpRRR4W3# +'giL;i +'giL&;i \\3dRu#i;i)z:(total, available) physical memory, cross-platform stdlib.Ndarwinz/usr/sbin/sysctlz-nz hw.memsizez/usr/bin/vm_statzpage size of (\d+)i@c3<"TFCp\P!V R2S4;o'gK'\SP^44xKE R#5i)z :\s+(\d+)\.N)researchr4group).0keymouts& r _ram_bytes..sLD8\YY3%{';SAAA,C OO8\s $A"Alinuxgetconf PAGE_SIZEi _PHYS_PAGES _AVPHYS_PAGES)rr)z Pages freezPages inactivezPages purgeablezPages speculative)r_rGrHsysplatformr4r!striprr&r8rdrerfsum startswithr'rA) r^r?availpage_mpagepagesmeminfo linux_ramrirjs @@r _ram_bytesr~s ! #D   $"3"333" <<8 # 2D,GMMOTTSTUEz QJE':..01#8#>/5s6<<?+5D8\DD19!LE/<  << " "7 + +)+G"3G< ($$79k2::d;GI}5::;dB  gz * * ?;@@qADHQQEE+|1/.,+ * Z  sr;H"&H"6%H"A,G< H"$H"7H"H".H" 3H"-H.H1 H"< H H" H H""H87H8ztuple[str | None] | None_smi_path_cachecV^8dQhRR/#r#r)rs"rrrs*rcT\e\^,#\P!R4pVf\PR8Xd\P P RR4p\V4R, R, \\P P RR 44R , R , R, 3p\R V4R4pMUVfR\PPR 4'd-\R4pVP4'd \V4pV3sV#)zAbsolute path to nvidia-smi, or None. PATH first (respects user overrides), then the driver's known install locations on Windows and WSL.Nz nvidia-smint SystemRootz C:\WindowsSystem32znvidia-smi.exe ProgramFileszC:\Program FileszNVIDIA CorporationNVSMIc3f"TF'qP4'gK\V4xK) R#5i)N)existsr)rgcs& rrk#_nvidia_smi_path..s?jHHJfc!ffjs11rmz/usr/lib/wsl/lib/nvidia-smi)rshutilwhichosr;environr5rnextrsrtrwrr)foundwindir candidates candidates r_nvidia_smi_pathrs"q!! LL &E }D m< L: %(8 8 0CD E" #%, -/? @  ?j?F 3<<227;;67      NEhO LrcV^8dQhRR/#)r rz'tuple[int, int, str, int | None] | Noner)rs"rrrsaa=arcv\4pVfR#VR,VR,VR,VPR43#)zb(total bytes, free bytes, name, optional packed PCI ID) from the shared nvidia-smi query, or None.N total_bytes free_bytesgpu_name gpu_pci_id)_cached_nvidia_gpu_queryr5)querys r _nvidia_vramrs: $ &E }  |!4eJ6GS_I` ``rz tuple[float, dict | None] | None_gpu_query_cacheg@c V^8dQhRRRR/#)r ttl_sfloatr dict | Noner)rs"rrrs++E++rc\P!4p\e'V\^,, V8d\^,#\4pVfVR3sR#^RIHp\ \\\P4;_uu_4\P!VRR.RRRR^ V!4RR 7pVP^8wg!VPP4'gVR3sRRR4R#\\ P"!VPP4P%4RR 74wrVrxrRp \ \4;_uu_4\'V^4p RRR4R VP4R \'V4^,R \'V4^,R\'V 4^,R\'V 4RV /p W3sV uuRRR4# +'giLt;i +'giM;iTR3sR#)uBOne nvidia-smi read shared by the budget probe and the hardware endpoint. Returns ``dict(gpu_name=, total_bytes=, free_bytes=, used_bytes=, gpu_util_percent=, gpu_pci_id=)`` or None when nvidia-smi is absent, fails, or is not an NVIDIA card. Cached for ``ttl_s`` (failures too — a missing smi must not spawn per poll). N)windows_hide_flagszS--query-gpu=memory.total,memory.free,name,pci.device_id,memory.used,utilization.gpuz--format=csv,noheader,nounitsTrrF)rrrrr creationflagsr)skipinitialspacerrr used_bytesgpu_util_percentr)time monotonicrrhermes_cli._subprocess_compatrrr&r8rTimeoutExpiredr returncoder rurcsvreaderr/r4) rnowexerrj total_mibfree_mibr;raw_idused_mibutilpci_iddatas & rrrs .. C#.>q.A(AE(I""  C {;@ ':z'@'@ A Ann g , .dWYXZ,.e = >>Q cjj&6&6&8&8 #T{  B A=A JJszz'')446 N=P9 T8 j ! !_F"  3y>R/ #h-2- #h-2- D &   ;/ B A" ! B A0T{ s3 AG !G 0A$G  G !A!G  G G  G0 cV^8dQhRR/#r rztuple[int, bool | None] | Noner)rs"rrrs9rc8^RIpRFpVPV4pM R#\\\4;_uu_4VP ^4^8wd RRR4R#VP 4pVPVPV4^4^8wd RRR4R#VP4p\VRR4;'g VPpV!VPV4V4^8wgVP^8:d RRR4R#RpVP 4pVPVPV4\V4^8Xd\VP4pVPV3uuRRR4# \dEK}i;i +'giR#;i)uD(allocator_total_bytes, integrated_or_None) from the CUDA driver API via ctypes against the driver's own DLL/SO — no toolkit, no subprocess, ~ms. INTEGRATED is the vendor's own unified-memory declaration; total is the pool the allocator will actually hand out (on carve-out devices, several times what nvidia-smi reports).NcuDeviceTotalMem_v2)z nvcuda.dllz libcuda.so.1z libcuda.so)rVCDLLr&rr6cuInitc_int cuDeviceGetr]c_size_tgetattrcuDeviceTotalMemr=cuDeviceGetAttribute_CU_DEVICE_ATTRIBUTE_INTEGRATEDbool)rVr;cudadevr?getter integratedattrs r_cuda_driver_poolrsb < ;;t$D = '> * * ;;q>Q  + *lln   FLL-q 1Q 6 + * !4d;TTt?T?T &,,u%s +q 0EKK14D + *#' ||~  $ $ T"$CS JMN Odjj)J{{J& + *     + * s5E5F 7F!#FsW7BA  F  "$((9??RG GKK 2<  mmnnc#h(89$"&TV\_`c`j`j\ksxz >>Q   ZZ#**,88:D&&t,Aq1771:",d22  ;     s$DADAD#$DD D/ cV^8dQhRR/#rr)rs"rrr`s  9 rc\P!4p\e \wrVfW, \8dV#\ 4;'g \ 4pW3sV#)zBest available allocator-side view, cached: a hit is permanent for the process, a miss retries after a short TTL (the engine binary can appear mid-session via a pane install).)rrr _POOL_NEGATIVE_TTL_Srr)rstampviews r_device_pool_viewr`sS .. C$'   s{-AAK   7 7"5"7D  Krz$dict[str, tuple[float, dict | None]]_accelerator_cachefreshFc V^8dQhRRRR/#)r rrrrr)rs"rrrqs$;rcZ\\4;_uu_4^RIHp\ 4pVeVP \ 9d RRR4R#\VP4\P!4rC\PV4pV'g;Ve7V^,fWE^,, \8dV^,uuRRR4#V!VPPVP 4p\VRRR7pWG3\V&VuuRRR4# +'giR#;i)aThe device a Vulkan/HIP engine will place layers on, or None (no such engine, probe miss). Mirrors llama.cpp's own placement: discrete devices whenever one exists, an integrated one only otherwise; among discrete devices the largest. Cached per engine binary like ``_device_pool_view`` (a hit lasts the process, a miss retries after the TTL): the hardware endpoint polls this every few seconds and each probe initializes the GPU driver. ``fresh`` skips the cache read for a launch-time free-memory answer. ) probe_devicesNc4VR,\8HVR,3#)typer?r)ds&r%_accelerator_device..sQvY/-I1W:,Vr)rhdefault)rr hermes_cli.local_runtime.devicesrrr_GPU_ENGINE_BACKENDSrrrrrr5rrmax)rrrrhrcacheddevicesdevices$ r_accelerator_devicerqs )  B#% >V^^3GG   v}}%t~~'7S#'',+1F#WXPY/\pJp!9    4 4fnnEW"V`de#&-3     s+D A5D AD D* c$V^8dQhRRRRRR/#)r smi_totalr4 ram_totalr int | Noner)rs"rrrs!33:rc\4pVfR#Vwr4Ve V'dV#R#V^8d@V\V\,48d%V^8dV\V\,48dV#R#)aThe real pool size when this NVIDIA device is unified memory behind a carve-out, else None. The driver's INTEGRATED attribute decides in BOTH directions when readable; only the attribute-less engine fallback needs the two numeric gates, both of which must hold. N)rr4_POOL_DISAGREEMENT_FACTOR_POOL_RAM_FRACTION)rrrpoolrs&& r_unified_pool_bytesrsi  D |D!t+t+A $#i2K&K"LLA $#i:L.L*M"M rrr lazy_readsTc 0V^8dQhRRRRRRRRRR R R /#) r baser4r?rrrrrrrrr)rs"rrrsAHHcH#HCH&H;?HKYHrc \^\V^\, ,44p\WQ^RV\P W4R7#)rrT)usable_vram_bytestotal_device_bytesram_available_bytesumarrtrr)rr4_UMA_HEADROOM_FRACTIONrrsrt)rr?rrrusables&&$$$ r _uma_budgetrsB C$: :;< =F F./THWZWcWc%/ HHrc V^8dQhRRRR/#)r own_bytesr4rrr)rs"rrrs K KC K Krc\4pVeVP'gR#\\\ VP \ ,44p\^\ VP4\^V4,V, 4#)aCommit the managed server may take on Windows right now, or None off Windows. Windows charges every GPU allocation to commit: the RAM Windows can see plus the page file. Memory carved out for the GPU is invisible to Windows, so a bigger carve-out leaves less commit for the model it was meant to hold. ``own_bytes`` is what the managed server holds now and frees before the new instance loads. A desktop margin stays free. N)r_rIr _MARGIN_FLOORr4rG_MARGIN_FRACTIONrJ)rr^margins& r_windows_commit_roomrse " #D |4000 D$5$58H$H I JF q#d++,s1i/@@6I JJrc V^8dQhRRRR/#)r model_bytesr4rr)rs"rrrs;;C;C;rc<\V^\, , 4#)zWhat Windows charges to commit for a model estimated at ``model_bytes`` of device memory: the estimate runs ~20% under the charge, which is the UMA headroom.)r4r)rs&rmodel_commit_bytesrs {a"889 ::rc(V^8dQhRRRRRRRR/#)r liveHardwareBudget | Nonerr4 residentsrrr)rs"rrrs+66266QT6]g6rc Ve VPMRpVeV^8:dR#\^V4p\V4p\\\^W2V,, 4V,4pV\ 8dV#^#)a~Size for llama-server's prompt cache beside a model of ``model_bytes`` (the estimate the window was planned with), or None when the engine default applies. Windows unified memory only: each loaded model's cache comes out of the same commit as the models, so each of the ``residents`` models the router may hold at once (its ``--models-max``) gets an even share of what they leave. N)commit_room_bytesrrmin_PROMPT_CACHE_MAX_PROMPT_CACHE_MIN)rrrroom model_commitcaches&&& rprompt_cache_bytesr(so&*%54 ! !4D |{a'Ay!I%k2L !3q$\1I*I#Ji#W XE..55A5rc$V^8dQhRRRRRR/#)r rrrr4rr)rs"rrrs&ZZ^ZZCZrc VP\^V4,\, p\VP\^\ V^\ , ,444#)zDevice memory a model may plan with on Windows unified memory and still leave each of the ``residents`` models the smallest useful prompt cache: the inverse of ``prompt_cache_bytes``.)r!rr$r"r r4r)rrshares&& rusable_beside_prompt_cacher,sK  " "c!Y&7 7:K KE t%%s1c%1?U;U2V.W'X YYrcV^8dQhRR/#)r rrr)rs"rrrs  D rcl\\4;_uu_4\4pVeVP\9uuRRR4#^RIHp^RIHpV!4PR4;'g/pT!VPR4;'gR4\9uuRRR4# +'giR#;i)zWhether a Vulkan/HIP engine runs the model: the serving or installed engine, else the one setup would install for ``local_runtime.backend``. False when the engine can't be resolved.Nr)resolve_backendrrrF) rrrrrrrrr/r5)rrr/rs r_gpu_engine_runs_the_modelr0s )  #%  >>%99   ;E&(,,_=CCw{{95??@DXX     s!B"(B"-B" B"" B3 c V^8dQhRRRR/#)r rrrrr)rs"rrrs&& & &rc^RIHpV!4Uu.uFq"P'dKVNK ppVUu.uF$q fKVPVR,8XgK"VNK& ppT;'g)VUu.uFq"P\ 8wgKVNK upp\ V4^8wgVf\4'gR#V^,P#uupiuupiuupi)aWhat Windows lets the integrated GPU allocate (Task Manager's GPU memory), or None. The adapter is the one the engine named, else the only non-NVIDIA hardware adapter when a Vulkan/HIP engine will run the model; the CPU engine (chosen, or the only build for a Snapdragon) keeps the model in system RAM. Not the engine's own figure: ggml-vulkan sums every memory heap of an integrated GPU, host-visible ones included, so it can exceed what Windows grants. )windows_gpu_adaptersN description) !hermes_platform.host.gpu_adaptersr3softwarer4 vendor_id_NVIDIA_VENDOR_IDr3r0 memory_bytes)rr3ahardwarenamedrs& r_windows_igpu_bytesr=sG/1D1a1HD b1Q1==FS`La;aQQE bSShSh++AR2R11hSJ :!7Q7S7S a= % %% E bSs&C C CCC.C Cplanningc V^8dQhRRRR/#)r r>rrrr)rs"rrrs9f9fd9f~9frc R\4wr\4p\V'd V^,M^V4pVe\P RV\ , V'dV^,\ , R R2MR4V'dTpM%V'dV^,V,MTp\ WF4p\YTV'd V^,MRV'dV^,R7#RR7#Vf\4pVeVR,\8XdnVR ,p\\\V\,44p \\^W, 4TV'dTMTR VR ,\P R 7#\ T\#V4;'gT4p \V'dT M \ W4V \%4R9R 7#Vwrr\\\V\,44p \\^V'dTMT V , 4TV'dTMTR V \P V R7#)uConstruct the budget per the source rules above. ``planning=False``: LIVE budget (free VRAM now) for launch-time fit and growth re-grants. ``planning=True``: CAPACITY budget (total minus margin) for catalog pricing and quant selection — pricing against live-free while a model was loaded made every row read as too large. The managed server unloads/relaunches itself, so capacity is real. Nziunified-memory NVIDIA device: allocator pool %.1f GiB (nvidia-smi carve-out: %s); budgeting from the poolz.1fz GiB unavailabler)rrrr?Fr4)r r r r rrt)r)r r r r rrtr)amdintel)r~rrloggerinfo_GIBr"rrrrrr4rrrsrtr=r )r>r ram_availvramunifiedrrrr?rrfreerrs$ r probe_budgetrKs&<I >D"T$q'q)DG  B dN+/tAw~c"$ ']  D D -1DGi'iDw%D4d4715d1gA A;?A A |$&  &.O"C7OEE4D,D(EFF!C5>4J_dCKiQZ&+f]6KVYVbVbd d91&9FFYG84T1Et&/k9I&IK K)-%E E,<$< = >F CXE4SY3Y,Z-2;Ci#hZd ffrrc$V^8dQhRRRRRR/#)r capacityrrr4rrr)rs"rrr/s" 7 7N 7# 7F[ 7rcRVP'db\V^8d \V4M^4pVfR#\VP\ V^\ , ,44p\WVR7#\4pVeV^,V^,reM5\RR7pVeVR,\8wdR#VR,VR,re\^WV, \^V4, 4p\VP\^WX, \, 44p\WR7#) aCapacity less what other programs hold on the card right now, or None when that can't apply. Capacity assumes other programs hold no more than the fixed margin. Beside a heavier desktop the capacity-sized window doesn't fit, and Windows pages part of the model to host memory without an error: a 32 GiB card with 4.5-6.3 GiB held by other apps decoded at ~24 tok/s at the capacity window, and at ~90 tok/s at a window sized from free memory. ``own_bytes`` is the device-memory estimate of what the managed server holds now and frees before the new instance loads. A stopped server's memory reads as free by the time its process has exited (measured on Windows: free memory was fully back at the first reading after exit). Unified memory: free commit on Windows (the model and its prompt cache must fit there), with ``own_bytes`` converted to the commit Windows charged for it, else None (its live budget is already free memory). None too when the device query fails, so callers keep the capacity plan. N)r r!T)rrr?rJ)r ) r rrr"r r4rrrrrr_LAUNCH_HEADROOM) rMrr%rrHr?rJrotherss &$ r launch_budgetrQ/s|||#YQR]$6y$AXYZ <X//TQAW=W5X1YZxTRR >D 1gtAwt%40 >VF^>Wovf~t EL3q)#44 5F ++SENEU4U-V WF 8 66ri@lli )vulkanhip>r.r,r-)rr))I__conditional_annotations____doc__ __future__r contextlibrrloggingrrdrrrsr dataclassesrpathlibrrr"hermes_cli.local_runtime.estimatorrhermes_platform.host.factsr getLoggerrNrDrFrrrOrr#r$rrrr8rrr __annotations__compilerr! frozensetr1r'rAr_r~rrrr_GPU_QUERY_TTL_Srrrrrrrrrrrr(r,r0r=rKrQ)rUs@rrcs#   <=0   8 $   #%(IMFM**OP CD F('Z-1)06a6:29 -=+\D >( <>8=4$H"H)-HBFH K; 6"Z &$9fU9fx 7 7 7r