߲jUdZddlZddlZddlZddlmZddlmZddlZddl Z ddl Z ddl m Z ddl mZmZmZmZmZmZddlmZddlZdaeed<daeed <dd Zdd lmZdd lmZddlm Z ddl!m"Z#m$Z%m&Z'm(Z)m*Z+m,Z-ddl.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5ejle7Z8eddZ9ddde:fdZ;dde:de:fdZ<ee>dZ>dZ?d e@fdZAd e@fdZBeBZCeeCd !ZDd"ZEd#e:d eFfd$ZGd%eHd eFfd&ZId'ZJd(ZKd#e:d)e d*e@d+e:d,e:d-e@d.e=d/eFd e fd0ZLdd1e:d)e d*e@d e fd2ZMdd3e d4ee:d e:fd5ZNd6ZOd7ZPd8ZQd9ZRd%eHd eFfd:ZSd;eeTdZVdeQddd?fd3e d4ee:d@e@dAee@dBeeTdCeFd e:fdDZWeXhdEZYeXhdFZZddGe:dHe:d eFfdIZ[dGe:dHe:d eFfdJZ\dGe:dHe:dKeee:efd eFfdLZ]d eFfdMZ^ dd1e:dNe:dOe:dPe@dQee:d ee:eff dRZ_dSee d dfdTZ`GdUdVeaZbGdWdXeZcd1e:dYee:dZeedd[eFd ecf d\ZedOe:d e:fd]Zfd^ecd;eTd e:fd_Zg dd1e:dNe:dYee:dZeedd ef d`ZhddadbeddHee:dce=ddee=d eTf deZidfe:dge:dhe:d edfdiZjdjZkekdkfdldmfZlekelddnfdoeRdpffZmdqelfdremfdsZndte:due:dfe:d eTfdvZoddweTdxe:fdyZpdte:due:dfe:dHee:dzee:eTedgeeqfd e:f d{Zr dd1e:dfe:dHe:dYee:dZeedd e:f d|Zsd eFfd}Ztd eFfd~ZuddlvmwZwmxZxddddddddddddidddddd1dNgddZydeqdeqd ee:fdZzdee:efded e:fdZ{ewjddeye{euddddddddddZ}dZ~d6Zde d ee:fdZde:d e:fdZdde d4ee:d e:fdZde:dYee:dedd e fdZ dde:dfe:dHe:dYee:d e:f dZddddddddddddNgddZdee:efded ee:fdZewjddeeetddddlZddlZddlZy)a[Vision tools: ``vision_analyze`` (image) and ``video_analyze``. Images resolve through :mod:`tools.image_source`, are normalized to a provider-supported format (:mod:`tools.vision_tools_image_prep`), then either attach natively to a vision-capable main model (multimodal tool-result envelope) or are described by the auxiliary vision LLM router. N)ThreadPoolExecutor)BytesIO)Path)Any AwaitableCallableDict NamedTupleOptional)urlparseasync_call_llmextract_content_or_reasoningreturncztt/ddlm}txs |jatxs |jayy)Nr)auxiliary_client)r ragentr)_auxs 7/home/ubuntu/.hermes/hermes-agent/tools/vision_tools.py_load_auxiliary_clientrs8!=!E2'>4+>+>'C'htGhGh$"F)get_hermes_dir) DebugSession)check_website_access)native_turn_duplicate record_embed release_embedrepeat_refusalresolve_repeat_capresolve_embed_target_bytes)&_VISION_MAX_VALIDATED_AGGREGATE_PIXELS!_VISION_MAX_VALIDATED_FRAME_COUNT_crop_image_region_determine_mime_type_image_exceeds_dimension_normalize_to_supported_image _validate_raster_image_decodable vision_toolsVISION_TOOLS_DEBUG)env_vardefaultkeyscZ ddlm}m}||dg|d|iS#t$r|cYSwxYw)zQ``auxiliary.`` from config.yaml; ``default`` when config is unavailable.r)cfg_get load_config auxiliaryr+)hermes_cli.configr.r/ Exception)r+r,r.r/s r_cfg_auxiliaryr3=s7:{}kJDJ'JJ s  **r)keycfd}|tj|djxsd}||S|td|S)uEnv var → ``auxiliary.vision.`` → None. Values that fail ``cast`` or fall below ``minimum`` are skipped in favor of the next source (a cap can never be disabled by a bad value).cV |}|k\r|SdS#ttf$rYywxYwN) TypeError ValueError)rawvalcastminimums r_acceptz%_read_vision_setting.._acceptIsA s)CosATA:&  s ((Nvision)osgetenvstripr3)r)r4r<r=r>r;s `` r_read_vision_settingrDFsIB "))GR(..08D 9C/3Mw~h/L'MMrHERMES_VISION_DOWNLOAD_TIMEOUTdownload_timeoutg>@i c tdttjdS#tt f$r%tdtj xsdcYSwxYw)zKUsable CPU count (``sched_getaffinity`` honors cpuset pinning), at least 1.r)maxlenrAsched_getaffinityAttributeErrorOSError cpu_countrr_detect_host_cpusrPbsP+1c"..q1233 G $+1blln)**+s'*1AAcDtddtd}|xs tS)udHERMES_VISION_MAX_CONCURRENCY → ``auxiliary.vision.max_concurrency`` → host cores (< 1 ignored).HERMES_VISION_MAX_CONCURRENCYmax_concurrencyrH)r=)rDintrP)r;s r_resolve_vision_cpu_workersrUjs% >@QSV`a bC  %#%%rz vision-encode) max_workersthread_name_prefixcKddl}tj}|jt|j |g|i|d{S7w)zZRun a sync encode/resize callable on the bounded vision CPU executor (never the LLM call).rN) functoolsasyncioget_running_looprun_in_executor_vision_cpu_executorpartial)fnargskwargsrYloops r_run_encode_on_cpu_executorrcwsK  # # %D%%&:jAd|dYwxYwwxYwdddd{7n#1d{7swYnxYwdddd{7n#1d{7swYnxYw|cS#t8$r}|} | |dz k\s |s>tC|s3t>jEd|| dzt#|ddd|sYd}~Ld| dzz}t>jGd|| dz|t#|dd|st>jGd|tIjJ|d{7Yd}~Dd}~wwxYww) uSSRF-safe streaming download with exponential backoff (2s/4s/8s). ``retry_all=False`` (images) retries only :func:`_is_retryable_download_error` errors — a 404/403 never succeeds on retry.r)atomic_replace)create_ssrf_safe_async_clientTparentsexist_okNmessagert)rfollow_redirects event_hooksGET)z User-AgentAccept)headerszcontent-lengthz too large (z bytes, max ).z.tmpwb missing_okz%Could not delete partial download: %sexc_inforHz*%s download failed after %s attempt(s): %sdz&%s download failed (attempt %s/%s): %s2zRetrying in %ss... _download_z3 exited retry loop without attempting (max_retries=)(utilsrrmrparentmkdirrangerrqr{stream_DOWNLOAD_USER_AGENTraise_for_statusrgetrTr9rjrd with_namenameuuiduuid4hexopen aiter_bytesrJwriter2unlinkrMloggerdebugrwrowarningrZsleep RuntimeErrorlower)rdr|r}r~rrrrrr last_errorattemptblockedclientrtcl declared_sizetmp_destination bytes_writtenfchunke wait_times r_download_mediars %>TD9J%?+> +*3/G%gi&8995$ZRfQgDh( ( s3GSY$Z'( ( ))+%%))*:;)/1CGtM!,1J$ }L|\e[ffg%hii.s8</G)0J>3 J%G, J%,I>G/B I>H$.H ?H HH =H %N)J>,J%/ G> :I>=G> >I>H H  H$H H$#I>$ I) .II) !I% "I) $I% %I) )I>, J%7I:8J%>J JJ J% J>J!J>%J7 +J.,J7 3J>;N> NAM= NA M=1M42M=7N=NN image_urlc TKt|||ddttdd{S7w)zCDownload an image with SSRF protection and error-class-aware retry.Imagezimage/*,*/*;q=0.8Fr~rrrrN)r_VISION_MAX_DOWNLOAD_BYTES_VISION_DOWNLOAD_TIMEOUT)rr|r}s r_download_imagers7 ; $7,6NZ_a aa as (&( image_path mime_typec|xs t|}d|dtj|jj dS)zK``data:;base64,...`` for a file (MIME from extension when not given).data:;base64,ascii)r#base64 b64encode read_bytesdecode)rrmimes r_image_to_base64_data_urlrsF  8,Z8D 4&!1!1*2G2G2I!J!Q!QRY!Z [ \\ri@i iP)z too largepayload413content_too_largerequest_too_largeexceedsz size limitclt|jtfdtdzDS)z;Detect if an API error is related to image or payload size.c3&K|]}|v ywr7rO.0hinterr_strs r z'_is_image_size_error..s`4tw`)rinvalid_request)rjrany_SIZE_ERROR_HINTS)rors @r_is_image_size_errorrs-%j G `+- :>:>Fmax_base64_bytes max_dimension scale_out force_jpegc jj}|dzdzdz}d||kr+| t|stt |krSdt ffd }t } | tjd|Stjd |d z |d z |d z |xs td k(xr| } | rd nd \} } | j} | s| jdvr| jd} | rdnd}| j| jfx}d}dt dt ddffd }t#dD]}|dkDrt%t!| jdzdt%t!| jdzd}}|dk(r?| jdkDr0t%t!| jd| jz zd}nC|dk(r>| jdkDr/t%t!| jd| jz zd}||f|k(rn7| j'||f| j(} ||f}tjd||||duxs#t%| j| j|k}|D]}t+}| j,|fd| i|ind|id| dt/j0|j3j5d}t ||kso|srtjd t |d z || j| j|| j| j|ccS||Stj7d!|d z t |d z || j| j|S#t$r'}tjd||cYd}~Sd}~wwxYw)"u]Base64 data URL, progressively downscaled with Pillow while over budget. Halves dimensions (aspect-preserving, 64px floor) up to 4 times; JPEG also walks a quality ladder (85/70/50) per step. Without Pillow, or if it still doesn't fit, returns the best attempt (or raw bytes) and lets the caller apply the size check. ``max_dimension``: force a downscale above this long edge even when bytes fit (Anthropic's 8000px cap is independent of bytes). ``force_jpeg``: re-encode PNG as JPEG when resizing — halving PNG dimensions destroys text legibility on dense screenshots. Args: max_dimension: If set, images whose longest side exceeds this pixel count are forcibly downscaled even if they're under the byte budget. Anthropic enforces an 8000 px per-side cap independently of the 5 MB byte cap. force_jpeg: Re-encode as JPEG even for PNG input when a resize is needed. History-reuse embeds (#92699) opt in so a text-heavy screenshot keeps its readable resolution and shrinks via JPEG quality instead. Images already under both caps are returned unchanged (still PNG). rNrrc&xs tS)Nr)r)data_urlrrsr_rawz&_resize_image_for_vision.._rawasU4Z9UUru;Pillow not installed — cannot auto-resize oversized imagezcImage file is %.1f MB (estimated base64 %.1f MB, limit %.1f MB, max_dimension=%s), auto-resizing... image/png)PNGr )JPEGz image/jpegz)Pillow cannot open image for resizing: %s>LRGBrr7)UFrwhcR$||fk7rjdd||yyy)NrrH)rrrr)update)rr orig_dimsrs r _record_scalez/_resize_image_for_vision.._record_scalezs=  aVy%8    ! )A,Z[hi  j&9 rrg?@zResized to %dx%d (attempt %d)formatqualityrrrz4Auto-resized image fits: %.1f MB (quality=%s, %dx%d)z=Auto-resize could not fit image under %.1f MB (best: %.1f MB))statst_sizer$rrJrjrrinfor#rr2modeconvertwidthheightrTrrIresizeLANCZOSrsaverrgetvaluerr)rrrrrr file_size estimated_b64r ris_png pil_formatout_mimeimgexc quality_steps prev_dims candidaterrnew_wnew_hdims_okqbufr rs`` ` @@r_resize_image_for_visionr7Cs(!))I]q(3.MH((!&>z=&Y,Z9M x=, ,OV#V % 'E } QRv  KKu[)=K+H K0-A;/ ; K ^T^P^F39/?UJjj$ chhl2kk% %GSUW:X5E{syy1}C b399n =>C"aC R#**_ =>Cu~***eU^U]];CI KK7w O4'V3syy#**+E+V !A)C CHHS U UaiiQR^ Uz&2B2B3<<>2R2Y2YZa2b1cdI9~!11g R Nk:Asyy#**Vcii4   !!.v  NNR#{3S^{5SU#))SZZ( U  ?Ev sN.. O7OOO> openai-chat azure-openai openai-codex google-vertexanthropic-directanthropic-vertexnousclaudeopenaivertexbedrock anthropic openrouter> google-geminigoogle-vertex-geminigeminigoogleprovidermodelc@ ddlm}|||S#t$rYywxYw)uHard veto: the provider's ``ProviderProfile`` declares ``supports_vision_tool_messages=False`` — images are accepted in user messages but list-type tool-result content is rejected with 400 (xiaomi/MiMo "text is not set"). ``supports_vision`` alone must not override this, or the multimodal tool-result envelope 400s every turn and the image never enters context (#89981). r))routed_model_rejects_vision_tool_messagesF) providersrLr2)rIrJrLs r_profile_rejects_tool_mediarNs)G85II s  ct|tr|jjnd}|r t ||ry|t vry|t vrDt|tr|jjndtfddDS ddlm }||}|duxrt|jS#t$rYywxYw) a Whether provider+model accepts image content inside a tool-result message. Unknown providers are False (caller falls back to aux-LLM text) unless their ``ProviderProfile`` declares ``supports_vision``; ``supports_vision_tool_messages=False`` is a hard veto.r?FTc3&K|]}|v ywr7rO)rtagms rrz2_supports_media_in_tool_results..sV3!8Vr)zgemini-3z gemini-pro-3zgemini-flash-3r)get_provider_profileN) rirjrCrrN_TOOL_RESULT_MEDIA_PROVIDERS_GEMINI_PROVIDERSrrMrSboolsupports_visionr2)rIrJprSprofilerRs @r_supports_media_in_tool_resultsrZs%/x$= 2A +Au5 (( %/s%;EKKM   !V'UVVV2&q)d"DtG,C,C'DD s(C C  C cfgcZt||ryt||ryddlm}||||duS)uwOne gate for both native lanes — the ``vision_analyze`` fast path and the ``computer_use`` capture route (#115248): the profile's ``supports_vision_tool_messages=False`` veto first, then either the provider's tool results are known to carry media or the capability lookup (config override → catalog → probes → profile) attests the model as vision-capable.FTr)_lookup_supports_vision)rNrZagent.image_routingr])rIrJr[r]s r_accepts_tool_result_imagesr_s5 #8U3&x7; "8UC 8D @@rc ddlm}m}ddlm}ddlm}|}|}|}||||dk7ryt|||S#t$r }tjd|Yd}~yd}~wwxYw) uTrue when image routing resolves to ``native`` AND the provider accepts images in tool results, or the user set the ``model.supports_vision`` override (escape hatch for custom/local providers). Any failure → False.r)_read_main_provider_read_main_model)decide_image_input_mode)r/nativeFz(Native vision fast-path check failed: %sN) agent.auxiliary_clientrarbr^rcr1r/r_r2rr)rarbrcr/rIrJr[r.s r#_should_use_native_vision_fast_pathrfsl P?1&( "m "8UC 8H D*8UC@@  ?Es6A A A/A**A/questionimage_data_urlimage_size_bytes scale_notecd}t|tr&|jr|d|jz }|r|d|z }dd|ddd|id gd |d z d d |dd|dddS)zMultimodal tool-result envelope. The text part is intentionally minimal (the model already has the question); ``text_summary`` is the fallback for providers without multimodal tool results.ulImage loaded into your context — you can see it natively now. Use your built-in vision to answer the user.z Question: z Note: Ttexttyperlrrd)rnrz,Image attached natively for the main model (.1fz# KB). Answer using built-in vision.N)r size_bytes native_vision) _multimodalcontent text_summarymeta)rirjrC)rrgrhrirj text_parts r _build_native_vision_tool_resultrys 7(C X^^%5~hnn&6%788 z*.. Y / ~/F GI;;Kd;RSV:WX, ,'o=M`de ggrpathcN| |jdyy#t$rYywxYw)NTr)rr2)rzs r_unlink_quietlyr|s4   KK4K (   s  $$ceZdZdZy)_ImagePrepErrorz=Raised by :func:`_prepare_image`; the message is user-facing.N)__name__ __module__ __qualname____doc__rOrrr~r~ sGrr~c@eZdZUdZeed<eeed<eed<e ed<y)_PreparedImagezHTemp image ready to encode; ``path`` is owned by the caller (delete it).rzrrrrN) rrrrr__annotations__r rjrTdictrOrrrrsR J 3-Orrtask_idregionvalidate_decodecKddlm}m}m} ||||d{}t dd} | jdd| d tjd z } tj| j|jd{|jt|ji} } } tjt | | d{\}} }|s|t |xsd || k7r't#| |} | j%j&} |r/t)t*| t,t.d{}|r t ||etjt0| || d{\}}}|s|t |xsd t#| |||j%j&} } } t5| | | | S7#|$r}t t ||d}~wwxYw7Z777#t2$r t#| wxYww)ujResolve → materialize → normalize → (validate) → (crop). Raises ``_ImagePrepError``. Unsupported formats (SVG, BMP) become PNG BEFORE encoding — an unsupported media_type baked into immutable history would 400 on every resume. The crop runs BEFORE any downscale so the region keeps the full resolution budget. On error no temp file is left.r)ImageResolutionErrorResolveContextresolve_image_sourcerNz cache/visiontemp_vision_imagesTr temp_image_z.imgzImage normalization failed.) offset_outzRegion crop failed.)tools.image_sourcerrrr~rjrrrrrZ to_thread write_bytesdatarrJr%r|rrrcr&r!r r" BaseExceptionr)rrrrrrrresolvedr.temp_dirrzrrrrnormalized_pathnorm_err decode_error cropped_path cropped_mimecrop_errs r_prepare_imagers ^]1-iPW9XYYn.BCH NN4$N/  DJJL>6 6D   D,,hmm <<<$,MM3x}}3Erk*D070A0AB_aegk0l*l'x .!("K.KL L d " D !"D,,J !<0$13Y"[[L%l33  9@9J9J"D&[:J4J 0L,</%h&G2GHH D !%1<ARARATA\A\ $D $j+ >>AZ 1c#h'S01 =+m[ 4J  s H GGGA'H  G''H 6G1G*A&G1;G-<3G1/G/0AG14H GG$ GG$$H *G1-G1/G11HH c>dt|dz ddtdz ddS)Nz2Image too large for vision API: base64 payload is r rp MB (limit .0fzw MB) even after resizing. Install Pillow (`pip install Pillow`) for better auto-resize, or compress the image manually.)rJ_MAX_BASE64_BYTES)rhs r_too_large_messagerDs= 15AA+Iv>N!i(G"$q(H)-HH)+2  mU;D H 9 %   HMM * I 7+IwX\]]H ; %x}}  OO 9:6 $hmm5IKK ~ !3 3z#3+!3CWdh$jjN>"%66!"4^"DeT H 9 %   HMM *  ) $/(>%00()<h>R>R>ZVZ[] H 9 %   HMM * G^ 7c#h6 6< H 9 %   HMM * E 7OKj I;SA23%8%HH H 9 %   HMM * I H 9 %   HMM * sAK0!I!:(K0#H5H6H:)I!#I$2I!I+I!I+I!.(K0AI!(K0H II#I$I!((K0II!I!I!! K*%J<KK(K0<KK)K--K0 min_timeoutmessagesdefault_timeoutrc|d}} tdixsi}|jd%tt|d| tdn|}|jdt|d}d|||d|rd|iSiS#t$rYwxYw) z``async_call_llm`` kwargs with ``auxiliary.vision.timeout`` / ``.temperature`` from config. Local vision models (llama.cpp, ollama) can take well over 30s, hence generous defaults (temperature 0.1); ``min_timeout`` lets video enforce a floor.g?r@r*rz-inf temperature)taskrrrrJ)r3rrIfloatr2)rrJrrrr _vision_cfgs r_aux_call_kwargsrs +C[G $Xr:@b ??9 % 1% I 67+J]vcnoG ??= ) 5 M :;K(;[b 3#( 33.0 33    sA&A<< BB user_prompt part_typer c$dd|dd||d|iigdgS)zJSingle user message: text + one ``image_url``/``video_url`` data-URL part.userrlrmrnrd)rolerurO)rrr s r_media_messagesrs6- 9uV^N_/`)b c ddr)402 insufficientzpayment requiredcreditsbillingzkInsufficient credits or payment required. Please top up your API provider account and try again. Error: {e}))does not supportznot support imagecontent_policy multimodalunrecognized request argumentz image inputzY{model} does not support vision or our request was not accepted by the server. Error: {e}))rrzThe vision API rejected the image. This can happen when the image is in an unsupported format, corrupted, or still too large after auto-resize. Try a smaller JPEG/PNG and retry. Error: {e}rH))rznot support videorrrz video input video_urlzThe model does not support video analysis or the request was rejected. Ensure you're using a video-capable model (e.g. google/gemini-2.5-flash). Error: {e}zbThe video is too large for the API. Try compressing or trimming the video (max ~50 MB). Error: {e}vision_analyze_toolvideo_analyze_tool)imagevideokindsourcec ft|dkDr|dddzn|}d|d|d|d|idddd d d d ||d d iS)Nrqz... parameters_urlrrJrorFanalysis_lengthr model_used _size_bytes)rJ)rrrrJrs r_debug_call_datarsf*-k*:S*@[#  &kF$t}fmVWeTy%):A|UW[V\\gThjk mmr call_kwargs empty_logcKt|tdi|d{}t|}|s1tj |ttdi|d{}|S7D7 w)zAux vision LLM analysis text, retrying once on empty content (reasoning-only response). ``response``: an already-made first call (the image path handles size-error retries itself).NrO)rr rrr)rrrtanalysiss r_call_vision_llmrsc'6+66+H5H y!/n6S{6S0ST O 71Ts!A&A"9A&A$ A&$A&stagec Kt|\}t|ts| t|nd}t||||g}dtdtffd } ddlm}|rOtdd |D]=} | s| js | jtjd |?Stjd||ddtjd|dd|||d{\} } | r t| nd} tjd|j!| | xsd|d} d | rd| d| n| d}| r| |d<j#d | |||D]=} | s| js | jtjd |?S#t$r#} tjd | d Yd} ~ ad} ~ wwxYw7#t$r"} tjd | d Yd} ~ d} ~ wwxYw#t$r}d|dt|}tj%d|d t|j't)fd|Dd|d}|j+|| } |d!<|d || d"cYd}~|D]k} | s| js | jtjd |?#t$r"} tjd | d Yd} ~ ed} ~ wwxYwSd}~wwxYw#|D]k} | s| js | jtjd |?#t$r"} tjd | d Yd} ~ ed} ~ wwxYwwxYww)#utAux-LLM analysis skeleton for image/video: interrupt check → ``stage`` → JSON result. ``stage(user_prompt, debug_call_data, temp_paths)`` returns ``(analysis, scale_note)``; temp files it appends to ``temp_paths`` are deleted in ``finally``. Returns JSON ``{"success": bool, "analysis": str}`` (``analysis`` carries the error explanation on failure). Nr?resultrctjtjtj|ddS)NrF)indent ensure_ascii)_debuglog_callr&jsondumps)rdebug_call_data tool_names rfinishz_run_analysis..finishs- ?3 zz&??rrrrFrzCleaned up temporary %s filez#Could not delete temporary file: %sTrzAnalyzing %s: %s<zUser prompt: %srz%%s analysis completed (%s characters)z-There was a problem with the request and the z could not be analyzed.[z] )rrrj)rrzError analyzing z: z%sc3RK|]\}}tfd|Ds| yw)c3&K|]}|v ywr7rOrs rrz*_run_analysis...s/RD/RrN)r)rhintstplrs rrz _run_analysis..s" SZUCC/RE/R,RS Ss''z" could not be analyzed. Error: {e})rrJro)rror)_ANALYSIS_KINDSrirjrrrrrexistsrrrr2rrrJ capitalizerrornextr)rrrrJrrules temp_pathsrrrz cleanup_errorrrjrrr error_msgtemplaterrrs @@@r _run_analysisrsp't,Iu k3 '*5*Ac+&r &tV[%HOJ@t@@ h2  mU;. hD hKKMLL!?F  h-  &fSbk: %{4C'89%*;%TT*+3#h- ;T__=NP_`l!NtfTkl!z:,b /K_gh #-F< t_Mf~ hD hKKMLL!?F  h !hNN#H-bfNggh3 U2!hNN#H-bfNggh!  T&tfBs1vh7  T9t 4a&,,. S5 S;D6Ae fh??Qe?4#, %)RSS hD hKKMLL!?F hNN#H-bfNggh  h T hD hKKMLL!?F hNN#H-bfNggh  hs)ANH .N6N&F*.N1AH 1G2A5H 'N/N&G'N* G3G NGNH  H$H<NHN L A>LL LNN/&KN LK<7N<LNL  LNN)&MN M; M6 1N6M; ;NNcKdtdtdtdtffd }t d||d{S7w)zDescribe an image (URL, local path, data: URL) with the auxiliary vision LLM. ``user_prompt`` is pre-formatted by the caller. Temp images live under $HERMES_HOME/cache/vision/.rrrrcKt  dd{}|j|jtj d|j dz tj dt t|j|jd{}tj dt|dz i}t|tkDr:t||d{}t|tkDrtt||j |d<t|d |}tj d t| d }t! t#di|d{}t+|d|d{} | t-|xsd|j.xsdfS77 77A#t$$r} t'| rt|t(kDstj d t|d z t(d z t||d{7}||dddd d<t#di|d{7}Yd} ~ d} ~ wwxYw7ƭw)NFrzImage ready (%.1f KB)rozConverting image to base64...rz#Image converted to base64 (%.1f KB)rirz%Processing image with vision model...g^@zYAPI rejected image (%.1f MB, likely too large); auto-resizing to ~%.0f MB and retrying...r rrurHrdz0Vision LLM returned empty content, retrying oncerO)rrrzrrrrrcrrrJrrr9rrrrr r2r_RESIZE_TARGET_BYTESrrr)rrrrrhrrrrt_api_errrrrJrrs rrz"vision_analyze_tool..stages' 7FTYZZ(--( +X-@-@4-GH 34: %x}}  OO 93~;NQU;UV ~ !2 2#3Hk#JJN>"%66 !3N!CDD.6.A.A*+"6;G ;<&x>   ;+:k::H* KXWW*;+>$@T@T@\X\]]]?[ O K; ;(2s>7JMa7a KKkN#{35I[5Y [$4Hk#JJJN.Is) DF * 0R 0 34 ? srIautoN)rerrrrs @rcheck_video_requirementsr<sGV    KMPZ\bOcJd     s3<c.txs tS)u~Image gate (``vision_analyze``, ``browser_vision``): an aux vision client OR the native fast path. Both handlers attach pixels straight to a vision-capable main model, so a main model on a provider the aux resolver does not know (OAuth, local vLLM) must not hide a working tool (#47149). ``video_analyze`` keeps the aux-only gate — its handler has no native path. )rfrrOrrcheck_vision_requirementsrNs / 0 N4L4NNr)registryrvision_analyzeuLoad an image into the conversation so you can see it. Call it any time the user references an image — then answer from what you see.objectstringz>Image URL (http/https), local file path, or data: URL to load.)rn descriptionz)Your question or request about the image.arrayrnintegerruOptional [x1, y1, x2, y2] crop in ORIGINAL-image pixel coordinates, applied before any downscaling — the crop keeps full resolution. Load the full image first, then re-call with a region to zoom into small text or fine detail.)rnitemsminItemsmaxItemsr)rrgr)rn propertiesrequired)rrrsectionsenv_varsc|D]F}t|d}|st|jrt|jcSntdd|DDdS)zFirst non-empty ``auxiliary.
.model`` from config.yaml, else the first non-empty env var (legacy override), else None.rJc3&K|] }|s| ywr7rO)rvs rrz(_configured_aux_model..sNqANsc3dK|](}tj|dj*yw)r?N)rArBrC)rrs rrz(_configured_aux_model..s$H!RYYq"-335Hs.0N)r3rjrCr)r%r&section_vmodels r_configured_aux_modelr-sd '2 7|!!#7|))++   NHxHNPT UUrr`rc\K|jdd|jdd|jd}}}|jd}tr,tjdt ||||d{Sd|}t d d }t |||||d{S7-7w) Nrr?rgrrz vision_analyze: native fast path)rrz]Fully describe and explain everything about this image, then answer the following question: )r@)AUXILIARY_VISION_MODEL)rrfrrrr-r)r`rrrgrr full_promptrJs r_handle_vision_analyzer1s"&((;";TXXjRT=UW[W_W_`hWixIffYG+, 67+IxY_``` ""* - "+/J KE$Y UG\bc cca ds$A8B,:B(;(B,#B*$B,*B,r@Tu👁️)rtoolsetschemahandlercheck_fnis_asyncemoji video/mp4z video/webmz video/movz video/mpeg).mp4z.webmz.movz.aviz.mkvz.mpegz.mpg video_pathc\tj|jjS)z7Video MIME type from extension, or None if unsupported.)_VIDEO_MIME_TYPESrsuffixr)r:s r_detect_video_mime_typer>s"  !2!2!8!8!: ;;rr=c fd|ddjttjS)NzUnsupported video format: 'z'. Supported: r)rsortedr<r,)r=s r_unsupported_video_formatrAs. (tyyPaPfPfPhIi?j>k llrc|xst|xsd}d|dtj|jj dS)Nr8rrr)r>rrrr)r:rrs r_video_to_base64_data_urlrCsK  J/ ; J{D 4&!1!1*2G2G2I!J!Q!QRY!Z [ \\rrrc Kddlm}m}m}m}|j d}t tjj|}|xsdjj} t| xr| jd } |s| rtjd|t |j j} | t"vrt%t'|  ||||d d {} t)d d }|j+dd|dt-j.| z }|j1| j2|j5||S|j7r/ddlm}|t=|tjd||St?|d {rrtA|}|rtC|dt)d d dt-j.dz }|j5|tE||dddtFddd {|St%d7=#|$r} t%d | | d } ~ wwxYw772w)uLocal video path for a terminal-backend path, local file, or HTTP(S) URL. Only files created here are appended to ``temp_paths`` — never user-provided paths. Terminal-backend reads use the shared media resolver with ``permitted=("video",)`` — the exact pipeline vision_analyze uses (media-cache host reads, bounded in-sandbox exec-read, credential-read guard, 50MB cap).r)rr_is_local_terminal_backendrzfile://r?)rfrgrz-Reading video source via terminal backend: %sr)r) permittedNz,Could not read video from terminal backend: z cache/videotemp_video_filesTrterminal_video_)raise_if_read_blockedzUsing local video file: %sr temp_video_r9rVideozvideo/*,*/*;q=0.8gN@rzKInvalid video source. Provide an HTTP/HTTPS URL or a valid local file path.)$rrrrEr removeprefixrrArz expanduserrCrrVrkrrr=r<r9rArrrrrrris_fileagent.file_safetyrIrjrnrrqr_MAX_VIDEO_BASE64_BYTES)rrrrrrErr local_pathlowered path_liker=rr.rrzrIrs r_materialize_videorTsC  # #I .Fbgg((01JB%%'--/GW Zg&8&89Y&Z"ZI % 'I CYOf$$**, * *6v>? ? \1>':jRRH"-1CDtd3ODJJL>&BB '$ ;c*o. 0)< &y 111&y1 !')"45 5m-?@[QUQ[Q[Q]P^^bCcc$ tQGH=B6I8I9A/I(I)I:H==IIIIIcKdtdtdtdtffd }t d||d{S7w)zEAnalyze a video via multimodal LLM. Returns JSON {success, analysis}.rrrrc<Kt |d{}|jj}|dz }tj d|t |}|st t|j|tkDrtjd|t||}t|tkDr't dt|dz ddtdz dd ||d <t|d |}t| d d } t!| dd{} | dfS77 w)Nr zVideo ready (%.1f MB)u,Video is %.1f MB — may be slow or rejectedrz+Video too large for API: base64 payload is rprrz+ MB). Compress or trim the video and retry.video_size_bytesrgf@rz#Empty video response, retrying once)rTrrrrr>r9rAr=_VIDEO_SIZE_WARN_BYTESrrCrJrPrrr)rrrtemp_video_pathrW video_size_mb detected_mimevideo_data_urlrrrrJrrs rrz!video_analyze_tool..stages/ 29gz RR*//199(K8  +];/@ 67M7MNO O 4 4 NNI= Y2?m\ ~ !8 8=c.>QU`>abe=fg1[A#FG89: :/?*+"6;G&x5Q )+7\]]~'S$^s"DDC9DD DDrNr )rrrJrrs` `` rrrsCC$DU*w ;uM MM Ms 3?=? video_analyzeu?Analyze a video from a URL or local file path using a multimodal AI model. Sends the video to a video-capable model (e.g. Gemini) for understanding. Use this for video files — for images, use vision_analyze instead. Supports mp4, webm, mov, avi, mkv, mpeg formats. Note: large videos (>20 MB) may be slow; max ~50 MB.z5Video URL (http/https) or local file path to analyze.zpYour specific question about the video. The AI will describe what happens in the video and answer your question.)rrgc |jdd|jdd}}d|}tdd}t||||jdS) Nrr?rgzFully describe and explain everything happening in this video, including visual content, motion, audio cues, text overlays, and scene transitions. Then answer the following question: )rr@)AUXILIARY_VIDEO_MODELr/rr)rr-r)r`rrrgr0rJs r_handle_video_analyzer`-s`((;3TXXj"5MxI ??Gj J ""57j kE ieRVVIEV WWrru🎬)rNr7)r)r?)NN)NNN)rrrZrconcurrent.futuresriorloggingrArpathlibrtypingrrrr r r urllib.parser rrr rrrhermes_constantsrtools.debug_helpersrtools.website_policyr!tools.vision_tools_history_budgetrrrrrrrrrrrrtools.vision_tools_image_prepr r!r"r#r$r%r& getLoggerrrrrjr3rDrrrrTrPrU_VISION_CPU_WORKERSr]rcrVrnr2rwrr{rrrrrr rrrrrr7 frozensetrTrUrNrZr_rfryr|r9r~rr rrrrrr_BILLING_HINTS_IMAGE_ERROR_RULES_VIDEO_ERROR_RULESrrrrrrrrtools.registryrrVISION_ANALYZE_SCHEMAr-r1registerr<rPrXr>rArCrTrVIDEO_ANALYZE_SCHEMAr` contextlibsys threadingrOrrrys 1 GG! $(c(i,,5&&&   8 $ n.B C(,# N# NC N00PRdfkl##.+3+&S& 23)# d((( d9 ZMm MmMm.1MmMm!Mm.1Mm16 VV(3-V03V-5c]V*2$V+/ V&NEH&NR $ $O4O0   !_ !J   ),  .!*-3(V VE VU Vx} VdtCH~dSdSd"   " &         +)<<#< mcmcm]$]8C=]TW] ,d,dhsm,dQU,dZ^,d`SWNN!$N-0NBJ3-N[^N8  ?!V ! R  !*- 4XS#XXcXinX    ! %   r