߲j4UdZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZej(dZd d d Zeed <d ZdZdZddde ee fdededededede ede efdZddde de eddfdZ ddd de dede ed!eddf d"Z!deddfd#Z"defd$Z#d-d%efd&Z$defd'Z%d.de efd(Z&dedefd)Z'd*Z(d ddedefd+Z)de eddfd,Z*y)/aSandbox lifecycle for the terminal tool: idle reaping, teardown, manual/atexit cleanup, and the lazy ensure_task_env bring-up. The env cache dicts and locks stay in tools.terminal_tool (tests patch them there) and are read through it at call time. Split out of ``tools/terminal_tool.py``; every public/patched name is re-imported there, so ``tools.terminal_tool.`` keeps resolving (and monkeypatching) as before. N)Path)AnyDictOptional)_get_scratch_dir)_container_config_from_config_ssh_config_from_config)_quietztools.terminal_toolgF) timestampresult_disk_usage_cachegr@cTtjttdz S)Nzhermes-*)globstrrB/home/ubuntu/.hermes/hermes-agent/tools/terminal_tool_lifecycle.py_scratch_pathsr$s 99S)+j89 ::rcTddlm}tjtdz t kr tdS d}t D]j}t|jdD]K}|jstd|t5||jjz }dddMl|d z }||kD}|rtjd ||tjtd<|td<|S#1swYxYw#t $rtj#d d YywxYw)zNTrue when hermes scratch dirs exceed the warning threshold (cached, advisory).r)DISK_USAGE_WARNING_THRESHOLD_GBr r *zCould not stat file %sexcNi@z\Disk usage (%.1fGB) exceeds threshold (%.0fGB). Consider running cleanup_all_environments().zDisk usage warning check failedTexc_infoF)tools.terminal_toolrtime monotonicr _DISK_USAGE_CACHE_TTLrrrglobis_filer OSErrorstatst_sizeloggerwarning Exceptiondebug)r total_bytespathftotal_gbexceededs r_check_disk_usage_warningr.(s!C ~~+K88;PP ** "$ 8D$Z%%c* 899; 8!I8#qvvx'7'77 88 8 8 ),==  NNy#%D F)-)9+&&.(#88  6 Fs0%6G*62T5J85T )& 1Z^!7X  r force_removeenvrAreturnc2t|drQ|>dtj|jjvr|j|y|jyt|dr|j yt|dr|j yy)a}Tear down one environment via cleanup()/stop()/terminate(), whichever it has. ``force_remove`` is forwarded to ``cleanup()`` only when given and the backend's signature accepts it (``DockerEnvironment``, issue #20561; other backends don't). Shared by ``cleanup_vm``, the idle reaper and the prompt-time backend probe so the signature check lives in one place. cleanupNrAr@stop terminate)hasattrinspect signaturerE parametersrFrG)rBrAs r _cleanup_envrLTsrsI  #':K:KCKK:X:c:c(c KK\K 2 KKM f   k "  #rz,Cleaned up inactive environment for task: %srAdone_msgrNc  t||tj||y#t$r^}t |}d|vsd|j vrtjd|ntj d||Yd}~yYd}~yd}~wwxYw)ux``_cleanup_env`` plus outcome logging. A 404/"not found" error means the sandbox is already gone — logged at info.r@404z not foundz*Environment for task %s already cleaned upz-Error cleaning up environment for task %s: %sN)rLr%infor'rlowerr&)rBr5rArNe error_strs r _teardown_envrUgs{XS|4 Hg& XF I  0A!A KKDg N NNJGUV W W OXs#& B ABB c@ ddlm}||y#t$rYywxYw)zZInvalidate the file_ops cache entry so ShellFileOperations can't reference a dead sandbox.r)clear_file_ops_cacheN)tools.file_toolsrW ImportError)r5rWs r_clear_file_ops_cacherZus$ 9W%    s  cddlm}m}m}m}m}|5|j |d}|j |dddd|5|j |ddddS#1swY(xYw#1swYSxYw)uPop *task_id* from the env cache, activity map and creation locks; return the env (or None). Callers run the (slow) teardown OUTSIDE the lock — Modal/Docker teardown can block 10-15s and would stall every concurrent terminal/file tool call.r_active_environments_creation_locks_creation_locks_lock _env_lock_last_activityN)rr]r^r_r`rapop)r5r]r^r_r`rarBs r_unregister_envrc~sw  *"&&w57D)* +GT*+ J **+ Js%AA)A&)A3lifetime_secondsc ddlm}m}m}m}m}t j } ddlm}t|jD]}|j|s|||< |5t|jD cgc]\} } || z |kDs| } } } | D cgc]} | |j| df} } | D]} |j| d|5| D]} |j| d dddddd D]\}} |  t|t!| |!y#t$rYwxYwcc} } wcc} w#1swYPxYw#1swYTxYw)zOClean up environments that have been inactive for longer than lifetime_seconds.rr\)process_registryN)rr]r^r_r`rartools.process_registryrflistkeyshas_active_processesrYitemsrbrZrU)rdr]r^r_r`ra current_timerfr5tlaststale envs_to_stoprBs r_cleanup_inactive_envsrqsg99;L ;N//12 7G44W=*6w' 7 -"&~';';'=">iwq$,QUBUXhBhiiHMN1044Q=>N N (A   q$ ' ( ! - -##At, - - -%(  ? !' * #w '(    jN - - --s_3D#D#"E ?D2D2E D86E D=.E # D/.D/2 E =E E  Ecddlm}m}m}||}|5|j |xs|j |cdddS#1swYyxYw)z9Return the active BaseEnvironment for *task_id*, or None.r)r]r`_resolve_container_task_idN)rr]r`rsget)r5r]r`rslookups rget_active_envrvsM__ ' 0F U#''/T3G3K3KG3TUUUs $AA c :ddlm}m}m}m}m}m}m}m}m } m } m } |} | d} | dk(ry||}t|}|$|5tj||<ddd|S| | | || }| |5|j|tj }ddd5t|}| |cdddS t#| | || d| d||| |}|5|||<tj||<dddt&j+d | |dd |cdddS#1swY|SxYw#1swYxYw#t$$r-}t&j)d | |dd |Yd}~dddyd}~wwxYw#1swYxYw#1swYyxYw) uLazily create and cache the sandbox env for *task_id* if none is active. Lets non-terminal callers (``tools.image_source`` reading container-only paths) bring the sandbox up on demand with the same machinery as the terminal tool. No-op on local. Returns the env, or ``None`` when local or when creation fails (best-effort; the caller's fail-closed path stays intact). :func:`terminal_tool` creates the environment on the first terminal command, but nothing else did — so under a non-local backend (ssh, docker, …) a session whose first action is ``vision_analyze`` on a container-only path hit "no active sandbox session" because the SSH/Docker handshake never ran (issue #62825). vision reads such paths inside the sandbox (see ``tools.image_source``), so it calls this to bring the env up on demand, reusing the same creation machinery as the terminal tool. r) r]r^r_r`_get_env_configrars_resolve_task_host_cwd _select_image_start_cleanup_threadresolve_task_overridesr1localNr3r4)r2r3r4r5r6z/Lazy %s environment init failed for task %s: %sz-%s environment lazily initialized for task %s)rr]r^r_r`rxrarsryrzr{r|rvr setdefault threadingLockr?r'r%r&rQ)r5r]r^r_r`rxrarsryrzr{r|r0r1effective_task_idexistingr2 task_locknew_envrs rensure_task_envrs  Fj!H727;/0H  <04 N, - < ($:7$CV LE T#../@)..BRS T !"34   ,6%=y)3D/@G <6= !2 304 N, - <  ; '+ 1 < TT  NNA+BQ/   < <#sfD3?%E.FE 'F)F#F3D=E  FE=0F=FFF FFcht|}|ytt|ddxs t|ddS)uTrue if *task_id*'s active env persists across turns. The agent loop skips per-turn teardown for these (persistent docker, daytona, modal, …); non-persistent backends are torn down at end of turn to prevent leakage, and the idle reaper handles the rest. Session-scoped docker containers count as persistent HERE: their lifetime is the session (removed by ``AIAgent.close()`` → ``cleanup_vm`` and the idle reaper). F_session_scoped _persistent)rvboolgetattr)r5rBs ris_persistent_envrs:  !C { .6\'#}V[:\ ]]rcddlm}d}t|jD]} t ||dz }tD]J}td|t 5tj|d t jd |dddL|dkDrt jd ||S#t $r#}t jd||dYd}~d}~wwxYw#1swYxYw) z3Clean up ALL active environments. Use with caution.r)r]zError cleaning %s: %sTrNz!Failed to remove orphaned path %sr) ignore_errorszRemoved orphaned: %szCleaned %d environments)rr]rhri cleanup_vmr'r%errorrr r"shutilrmtreerQ)r]cleanedr5rSr*s rcleanup_all_environmentsr s8G,1134M M w  qLGM 6 77 K 6 MM$d 3 KK. 5 6 66 { -w7 N M LL0'1tL L L M  6 6s#B,.C, C5CCC$ cTt|}t||yt|||dy)uManually clean up a specific environment by task_id. *force_remove* is forwarded to backends that accept it (currently only ``DockerEnvironment``). Default False matches session-lifecycle semantics: callers (``AIAgent.close()`` on TUI/gateway session teardown, the per-turn cleanup of non-persistent envs) must honor the user's persist-mode preference — stopping the container here would break the "ONE long-lived container shared across sessions" contract. Pass ``force_remove=True`` only for user-initiated teardown. The idle reaper calls ``env.cleanup()`` directly, so persist-mode idle envs are likewise no-op'd; only the orphan reaper at next startup reclaims them. Nz,Manually cleaned up environment for task: %srM)rcrZrU)r5rArBs rrrs0 ' "C'" { W<?rctddlm}m}m}m}||h}|r|j |g}|5|D]:}|j |d}|j |d|*|j|< ddd|D]&}td5|jddd(y#1swY5xYw#1swY?xYw)zDrop any cached env for *task_id* (and its collapsed key) after an infrastructure failure, so later calls don't reuse a dead connection.r)r]r`rarsNz&cleanup of degraded environment failed) rr]r`rarsaddrbappendr rE) r5r]r`rarsrievictedkeyrBs r_evict_environment_for_taskr6s 'w / 0D G $ $C&**35C   sD )s#  $$  < =  KKM   $$  s,B"B"B."B+.B7 )i,)N)+__doc__rloggingrIrrrpathlibrtypingrrrtools.environments.singularityrr=rr r>r getLoggerr%r dict__annotations__rrr.rintr?rrLrUrZrcrqrvrrrrrrrrrs   &&;.   0 1 ),u=4=;8KO cN&)58?BMP $SM9A$$>BcHTNd&MQcQ Xs XS X8D> X\_ XVZ X 3 4 S"(S(BUCU?Xc]?D ^s ^t ^,6;d.#4r