߲jMdZddlZddlZddlmZddlmZmZmZm Z ddl m Z ddl m ZddlmZddlmZd efd Zd ed ed ed e eeeeffdZd eded edeeefdeed eeeff dZd eded efdZy)ubrowser_vision helpers: Lightpanda pre-route, native provider vision, auxiliary-LLM screenshot analysis. Split out of ``tools/browser_tool.py``. Facade-owned state is read through ``_bt`` (``tools.browser_tool``, resolved per call) — no import cycle. N)Path)AnyDictOptionalTuple)cfg_get)origin)browser_tool_cloud) browser_tool_lightpanda_fallbackreturncRtj}|dSd|jdS)Nlocalzcloud ())_cloud_get_cloud_provider display_name)_cps >/home/ubuntu/.hermes/hermes-agent/tools/browser_tool_vision.py_vision_mode_labelrs/  $ $ &Ck7D1A1A0B!'DDeffective_task_idannotatescreenshot_pathcDtj}|dk7stj|sdd|fStjj d|rdgng}t j||tj}t j|tj}|jds4tjjd|jddd|fS|jd ijd d }|r~tjj|r_d dl}d d lm}|dd} | j'dd| d|j)j*dz } t-j.|| | }d|jd|fS)aMCapture the vision screenshot via the Chrome fallback when Lightpanda is the engine (it has no graphical renderer). Returns ``(prerouted, fallback_warning, path)``; on fallback failure ``prerouted`` is False and the caller takes the normal screenshot path (forcing Chrome) so the standard fallback metadata still applies. lightpandaFNzDbrowser_vision: pre-routing screenshot to Chrome (engine=lightpanda)z --annotatesuccessz7Lightpanda Chrome fallback vision screenshot failed: %serrordatapathr)get_hermes_dirzcache/screenshotsbrowser_screenshotsT)parentsexist_okbrowser_screenshot_z.pngfallback_warning)r_get_browser_engine_should_inject_engine_btloggerdebug_lp_chrome_fallback_screenshot_get_command_timeout_annotate_lightpanda_fallback_LP_VISION_FALLBACK_REASONgetwarningosrexistsuuidhermes_constantsr!mkdiruuid4hexshutilcopy2) rrrenginescreenshot_args fb_resultfb_pathuuid_modr!screenshots_dirpersistent_paths r_lightpanda_vision_prerouterCsg ' ' )F V%A%A&%IdO++JJ[\(0|nbO//0A?TWTlTlTnoI11)S=[=[\I == # TV_VcVcdkVlmdO++mmFB'++FB7G277>>'*3()<>STdT:).A(..BRBVBVAWW[,\\ Wo.) 12O CCrquestionresultlp_fallback_warningcddlm}m}m}ddlm}||d||d} |t ||| |jj} | jdi} t || d <|r|| d <|r,|jd ijd r |d d | d <| jd dd|j| d <| S)a5Multimodal tool-result envelope: the main model inspects the pixels itself. The embed is baked into history and re-sent every later turn, so apply the same proactive resize as vision_analyze's native path (skipped when already under both caps; without Pillow it fails open to the raw bytes). r)_EMBED_MAX_DIMENSION _build_native_vision_tool_result_resize_image_for_vision)resolve_embed_target_bytes image/pngT) mime_typemax_base64_bytes max_dimension force_jpeg) image_urlrDimage_data_urlimage_size_bytesmetarr&r annotations text_summaryr z Screenshot path: ) tools.vision_toolsrHrIrJ!tools.vision_tools_history_budgetrKstrstatst_size setdefaultr1strip) rrDrrErFrHrIrJrKdata_url native_resultrTs r_native_vision_resultr`4s M';9S9U6JW[]H5s??S^fDLFUFZFZF\FdFdfM  # #FB /D!/2D #6  FJJvr*..}=$Vn];]'4'8'8'L&MM_`o_p$q$w$w$yM.! rcddl}d|d}|j}|j|jd}d|}t j }tj jdt|d}d } dd l m } t| d d i } | jdt| d}| jdt| d} ddlm} d | |dd|ddd|idgdgd} |r|| d< | d$i| }dd"lm}||j0dj2j4xsd#j7S#t$rYwwxYw#t$r}ddlm}m}m}||rt||kDstj j+dt|dz |dz ||d}|| ddd d!dd<| d$i| }Yd}~d}~wwxYw)%u@One-shot aux vision-LLM analysis (not baked into history), secret-redacted. Full resolution first; on a size-related provider rejection the image is downscaled once and retried. ``auxiliary.vision.timeout/temperature`` — local vision models can take well over 30s, so the default timeout is generous. rNzCYou are analyzing a screenshot of a web browser. User's question: a" Provide a detailed and helpful answer based on what you see in the screenshot. If there are interactive elements, describe them. If there are verification challenges or CAPTCHAs, describe what type they are and what action might be needed. Focus on answering the user's specific question.asciizdata:image/png;base64,z/browser_vision: analysing screenshot (%d bytes)g^@g?) load_config auxiliaryvision)defaulttimeout temperature)call_llmusertext)typerkrQurl)rlrQ)rolecontent)taskrhrgmessagesmodel)_is_image_size_errorrJ_RESIZE_TARGET_BYTESzSVision API rejected screenshot (%.1f MB); auto-resizing to ~%.0f MB and retrying...irL)rMrqro)redact_sensitive_textr )base64 read_bytes b64encodedecoder)_get_vision_modelr*r+lenhermes_cli.configrcrr1float Exceptionagent.auxiliary_clientrirWrsrJrtinfo agent.redactrvchoicesmessageror])rrDrx vision_prompt_screenshot_bytes_screenshot_b64r^ vision_modelvision_timeoutvision_temperaturerc _vision_cfgri call_kwargsresponse_api_errrsrJrtrvs r _analyze_screenshot_with_aux_llmrUs $:&; <(224&&'89@@IO''89H((*LJJFL]H^_N 1km[(BO ??9 % 1";y#9:N ??= ) 5!&{='A!B 0);$] 3 x/@ A2   K+ G +*k*3 ("2"21"5"="="E"E"K!R!R!T UU7    +kk$X.3x=CW3W  mH 57K{7[ ]+O{SGO J"9-a0=eD*k*+s,AE=E EE GA7GG)__doc__r3r:pathlibrtypingrrrrr~rtools.browser_tool_originr r)toolsr rr r,rYrboolrCr`rrwrrrs  --%3.9ECE DD&*D=AD 4# $%D<%(48 cN19# #s(^B;Vd;Vc;Vc;Vr