߲js dZddlZddlZddlZddlmZmZmZdxZxZ xZ xZ Z ddl mZmZddlmZddlmZmZmZddlmZddlmZmZdd lmZmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ejPe)Z*d e+d e+fd Z,d e+d e-fdZ.d e/fdZ0dpde+d e+fdZ1de+fdZ2de+fdZ3dZ4dqde+de+d ee-fdZ5d e+fdZ6d ee+fdZ7d e+fdZ8d e+fdZ9d e-fdZ:d e-fdZ;d d!d"d#d$d%d&d'd(e;d) Z<e=e<Z>de+d e-fd*Z?d e@e+fd+ZAed,d-.ZBdrd/ZCdsd0e+d1e/d2ee+d ee+fd3ZDdtd5e+d6eEd e+fd7ZFd5e+d6eEd e/fd8ZGdud9eed:e+d;eeEd e+fd<ZHd e-fd=ZIe=d>hZJd e-fd?ZKdd@lLmMZMmNZNdAdBdCdDdEdFdGdHdIdJd4dKdLd5gdMdNZOdOdPdCdQdRdDidSd4dTdGdUdVdWdXd9gdMdNZPeMjdAdYeOdZeKeAd[d\]eMjdOdYePd^eKeAd_d`d\a ddblmRZRddclmSZSddlTZTddlUZUddlVZVddlWZWdddedfdgdhdidjdkdldmdn ZXdoZYy)va Generic web_search / web_extract tools over pluggable backends. Backend is selected during ``hermes tools`` (``web.backend`` in config.yaml; per capability via ``web.search_backend`` / ``web.extract_backend``). Every vendor implementation lives in ``plugins/web//provider.py`` and registers with ``agent.web_search_registry``; this module owns selection, safety gates, caching, keyless rescue, and the truncate-and-store result pipeline. Debug: ``WEB_TOOLS_DEBUG=true`` writes ``logs/web_tools_debug_.json``. N)ListAnyOptional)_is_tool_gateway_readycheck_firecrawl_api_key) DebugSession)NOUS_MANAGED_PROVIDERread_selectionselection_exists)async_is_safe_url)_rescue_eligible_rescue_search)_effective_char_limit _trim_results_truncate_resultsconvert_base64_images_to_links)_extract_safe_urls_merge_in_order_no_provider_error_resolve_extract_provider _result_entry_strict_selection_error_validate_extract_urlsnamereturnc ddlm}||}|tj|dn|xsdj S#t$rd}Y9wxYw)uResolve ``name`` via the config-aware env layer (``hermes config set`` values), then process env. Mirrors the SearXNG provider's ``_searxng_url()`` so that values set through Hermes' config/.env layer (``hermes config set``, ``hermes tools``) are honored here too — not just raw process-env exports. Without this, a config-only ``SEARXNG_URL`` (or any provider key) leaves the backend auto-detect cascade and ``check_web_api_key()`` blind to it. See #34290. r) get_env_valueN)hermes_cli.configr Exceptionosgetenvstrip)rrvals 4/home/ubuntu/.hermes/hermes-agent/tools/web_tools.py _env_valuer&#sQ3D!%(KRYYtR S ?R F F HH s< A  A c*tt|SN)boolr&)rs r%_has_envr*3s  4 !!cf ddlm}|jdxsiS#t$ricYSwxYw)zYLoad the ``web:`` section from config.yaml; always a dict (a null section yields ``{}``).r load_configweb)rr.getr r-s r%_load_web_configr17s61}  '-2-  s " 00backendkeycttj|xsdjjS)z?Lower-cased, stripped ``web.`` value ("" when unset/null).r)r1r0lowerr#)r3s r%_configured_backendr6@s-   " "3 ' -2 4 4 6 < < >>r+ func_namec ddlm}t|||S#t$r$}tj d||||cYd}~Sd}~wwxYw)zc``agent.web_search_registry.(*args)``, or *default* if it raised (registry never fatal).rNz%web provider registry %s%r failed: %s)agent.web_search_registryweb_search_registrygetattrr loggerdebug)r7defaultargs registry_modexcs r%_registry_callrBEsF8/w|Y/66  .$ 8MMr+ctdgS)z())``, or ``None`` if it raised (a broken provider is unavailable; *context* is appended to the debug log line, e.g. " during readiness check").rz!web provider %r.%s() raised%s: %sN)r)r;r r<r=)providerrLrMrArs r%_proberPYsV-GHf-/00 x2 8$QTUs A&AAc  t}|r |tk(rdS|Stdydtdfdtdfdtdfd td fd td fdtd xs tdfdt fdtdfdtdfdt ff }|D] \}}|s |cSt D]/}|jtvst|ds#|jcStxsdS)uShared web backend name. A stored ``web.backend`` is returned as-is — no availability probe, no fallback — so a broken selection surfaces the vendor's honest error rather than silently rerouting. The managed ``use_gateway`` selection also resolves to firecrawl with no ladder. Autodetect runs whenever no SHARED web selection was ever stored: per-capability keys (``web.search_backend``, ``web.extract_backend``) name only their own capability and never reroute the other (#113017). firecrawlr/tavilyTAVILY_API_KEY perplexityPERPLEXITY_API_KEYexa EXA_API_KEYparallelPARALLEL_API_KEYkeenableKEENABLE_API_KEYFIRECRAWL_API_KEYFIRECRAWL_API_URLsearxng SEARXNG_URL brave-freeBRAVE_SEARCH_API_KEYddgs is_available) r6r r r*r_ddgs_package_importablerKr_LEGACY_WEB_BACKENDSrP_keyless_backend) configuredbackend_candidatesr2 availablerOs r% _get_backendrkds) %&J(,AA{QzQe( 8,-.xH\?]0^ '( X012ZJ\A]4^ h23Tx@S7TU ,./)Xm=T1U x 6786C[C]:^ 1 N 34! == 4 4.9Y== !   ,,r+c tddlm}m}|r*|D] }t |}|t |ds|cSy#t $r }tjd|Yd}~yd}~wwxYw)zKeyless free-tier backend name, or None. Strictly the last autodetect rung so it never pre-empts a keyed backend. Discovery must run first: reachable from contexts that haven't loaded plugins (subprocess runs, delegate children).r)_keyless_preference_keyless_tier_enabledNis_keyless_availablez keyless fallback walk failed: %s) _ensure_web_plugins_loadedr9rmrnrGrPr r<r=)rmrnrrOrAs r%rgrgss >"$X "+- 3D9'F8=S,TK  > 7== >s(0A AAA A/A**A/c0tdxs tS)zaBackend for web_search: ``web.search_backend`` (strict, no probe) > ``web.backend`` > autodetect.search_backendr6rkrJr+r%_get_search_backendrts / 0 BLNBr+c0tdxs tS)zcBackend for web_extract: ``web.extract_backend`` (strict, no probe) > ``web.backend`` > autodetect.extract_backendrsrJr+r%_get_extract_backendrws 0 1 C\^Cr+c, ddl}y#t$rYywxYw)zXddgs is the only backend gated on package presence; single symbol so tests can patch it.rNTF)rc ImportError)rcs r%reres s  c< ddlm}|S#t$rYywxYw)Nrhas_xai_credentialsF)tools.xai_httpr|r r{s r%_xai_availabler~s%6"$$ s  ctdS)NrXr*rJr+r%rs 8M*r+ctdS)NrZrrJr+r%rr !34r+ctdS)Nr\rrJr+r%rrrr+ctSr()rrJr+r%rrs 02r+c@tdxstddDS)NrTc38K|]}t|dk(yw)rSN)r6).0ks r% z..s h! q !X - hs)r2rrrv)r*anyrJr+r%rrs$h/0i h7g hhr+ctdS)NrVrrJr+r%rrs (#78r+ctdS)Nr`rrJr+r%rrs x .r+ctdS)NrbrrJr+r%rrs (#9:r+ctSr()rerJr+r%rrs ,.r+) rWrYr[rRrSrUr_rarcxaic|xsdjj}|tvrdn t|}|t |dxsdSt j |}|r|SdS)uTrue when *backend* is usable — the single availability chokepoint. Non-legacy names delegate to the registered provider's ``is_available()`` (unregistered names fall through); built-ins use cheap probes. For plugin-registered backends (any name outside :data:`_LEGACY_WEB_BACKENDS`), availability is delegated to the provider's ``is_available()`` via the web_search_registry. This is the single chokepoint through which ``_get_backend``, ``_get_capability_backend``, and ``check_web_api_key`` all resolve availability — fixing custom-provider discovery for every caller at once (issues #28651, #31873, #32698). Built-in backends keep their cheap hardcoded probes below. rNrdF)r5r#rfrGrP_BUILTIN_AVAILABILITYr0)r2rOprobes r%_is_backend_availablersl}"##%++-G"66t tool sees it; extras are harmless for the not-logged-in.) rXrZrTrVr\r]r^FIRECRAWL_GATEWAY_URLTOOL_GATEWAY_DOMAINTOOL_GATEWAY_SCHEMETOOL_GATEWAY_USER_TOKENrJrJr+r%_web_requires_envrs  r+ web_toolsWEB_TOOLS_DEBUG)env_varcx ddlm}|y#t$r }tj d|Yd}~yd}~wwxYw)u]Idempotently run plugin discovery so the web registry is populated. Dispatch is reachable from contexts that never triggered discovery (subprocess agent runs, delegate children, scripts); without it a configured backend yields a misleading "No web ... provider" error. Every bundled web provider (brave-free, ddgs, searxng, exa, parallel, tavily, firecrawl, keenable) registers itself via ``plugins/web//__init__.py`` during plugin discovery. Tool dispatch can be reached from contexts that haven't already triggered discovery — subprocess agent runs, delegate children, standalone scripts, certain test paths — and without it the registry is empty and ``get_provider('firecrawl')`` returns ``None`` even when the user has ``web.extract_backend: firecrawl`` configured and ``FIRECRAWL_API_KEY`` set. See #27580. r)_ensure_plugins_discoveredz+Web plugin discovery failed (non-fatal): %sN)hermes_cli.pluginsrr r<warning)rrAs r%rprps5KA"$ KDcJJKs  949 call_namedebug_call_data error_msgc|tjd|||d<tj||tj |dSt |S)zhLog the call into the debug session; with *error_msg*, record it and return its ``tool_error`` envelope.Nz%serror)r<r=_debuglog_callsave tool_error)rrrs r% _finish_debugrsL T9%#,  OOI/ KKM$4?*Y*??r+querylimitc v ttt|dd}||dddddd} ddlm}|r td d Stdd l m }m }t}|r||nd}||jsN|E|rCtd r8td|x}|d<t!d|t#j$d |ddd S|}|d} d t'd| d} n/t(j+d|j,||t/|||} t1| j3dij3d g|d<t#j$| dd } t1| |d<t!d|| S#ttf$rd}YgwxYw#t4$r#} t!d|dt7| cYd} ~ Sd} ~ wwxYw)u Search the web via the configured backend. Returns a JSON string ``{"success": bool, "data": {"web": [{"title", "url", "description", "position"}, ...]}}`` (metadata only — use web_extract_tool for page content) or ``{"success": false, "error": ...}``. drrrNr) parametersr results_countoriginal_response_sizefinal_response_size)is_interrupted InterruptedF)success)get_active_search_providerrDr/searchrweb_search_tool)rrindent ensure_asciizDNo web search provider configured. Run `hermes tools` to set one up.z#Web search via %s: '%s' (limit: %d)datarrzError searching web: )minmaxint TypeError ValueErrortools.interruptrrrpr9rrDrtsupports_searchr rrjsondumpsrr<infor_memoized_searchlenr0r str) rrrrr_wsp_get_providerr2rO error_textfallback response_data result_jsones r%rrs CE A&,!&6XY"#AO c2  mU; ;"$k%'18$W-d  8#;#;#=G0@0G8OPXZa8bb _W5/Azzej"IRSbghh13H  ]H(-8J8U]8^_M KK=x}}eUZ [,XueDM+.}/@/@/L/P/PQVXZ/[+\(jjquM 14[1A-.'9G z "H c.CXY\]^Y_X`Aabbcs<E4F BF B+F 4F F  F8F3-F83F8cddlmm}m}dtt t fffd }|jj}|o|jj5|jj}|*|\}}|s|jj|ddd||S#1swYxYw)uTTL memo + single-flight around the paid vendor call (tools/web_result_cache.py); sits after every safety/config check. The provider is asked for the BUCKETED count so near-identical limits share an entry; the caller's count is sliced out. Only successful, non-rescued responses are cached — caching a rescue would make the one-shot ring fallback sticky for a whole TTL.r) bucket_limit search_memoslice_search_responserc j} j|}|j ds>tr3tjt |j dd|dfS|dfS#t$r9}tstjt ||dfcYd}~Sd}~wwxYw)NTrrrF)rr r rrrr0) fetch_limitresprArrrOrs r% _paid_searchz&_memoized_search.._paid_searchGs"5)  U??5+6D xx "'7'A!(--TXXgr5J1KUT_`bff fU{  U#H-!(--S5+NPTT T UsA00 B29.B-'B2-B2N) tools.web_result_cacherrrtupledictr)lookupr flight_lockstore) rOrrrrrr was_rescuedrs ``` @r%rr@s XW %d +   &&x}}eUCM  $ $X]]E5 A R'..x}}eUKM$-9^* {"%%hmmUE=Q  R ! 66 R Rs 'A CC urlsformat char_limitc HKt|\}}}}||S|||ddddddggd} tjdt|ggi} } }t ||D]J\} } t | d{r#|j | | j | O'w.&PZ[fjahi O ,i 7_9MN02B< 0/B JE3&s+++  %##E*&3U' U#   *,G & (#.s Dg  ) DsTr)rget_active_extract_providerNrFz3web provider registry availability check failed: %s)r6rf_WEB_CHECK_SKIPrrpr9rrr;rgrr r<r=)cb candidatesrrrOrAs r%check_web_api_keyr s 346<!1< 4Q18PQQRJ D DD"$e357R7TU H#&$(G?(Z$%1!(+ %=Q$  JCPs?B$B$ B)B)AB. B. B."B.. C7CC)registryr web_searchaSearch the web for information. Returns up to 5 results by default with titles, URLs, and descriptions. The query is passed through to the configured backend, so operators such as site:domain, filetype:pdf, intitle:word, -term, and "exact phrase" may work when the backend supports them.objectstringzThe search query to look up on the web. You may include backend-supported operators such as site:example.com, filetype:pdf, intitle:word, -term, or "exact phrase".)type descriptionintegerz3Maximum number of results to return. Defaults to 5.rr)rrminimummaximumr>r)r propertiesrequired)rrr web_extractu+Extract content from web page URLs. Returns clean page content in markdown/text (no LLM summarization — fast). Also works with PDF URLs (arxiv papers, documents) — pass the PDF link directly. Pages within the char budget (default 15000) return whole; larger pages return a head+tail window with a footer telling you the full text's saved file path and the read_file call to page through the omitted middle. Inline images appear as [IMAGE: alt] placeholders; real image URLs are kept as links. If a URL fails or times out, use the browser tool instead.arrayrz:List of URLs to extract content from (max 5 URLs per call))ritemsrmaxItemszOptional per-page character budget sent back (default 15000). Pages larger than this are head+tail truncated with the full text stored to disk. Raise it when you need more of a long page inline.i)rrr)rrr/c \t|jdd|jddS)Nrrrr)r)rr0r?kws r%rrs%txx/DDHHU\^_L`ar+u🔍i)rtoolsetschemahandlercheck_fn requires_envemojimax_result_size_charsc tt|jdtr|jdgddngd|jdS)Nrrmarkdownr)r)r isinstancer0listrs r%rrsD/$.txx/?$FRa BPZ88L) r+Tu📄) rrrrrr is_asyncr!r")Dict) TYPE_CHECKING)tools.web_tools_truncateDEFAULT_EXTRACT_CHAR_LIMIT)zplugins.web.firecrawl.provider Firecrawl)r*MAX_STORED_TEXT_CHARS)tools.managed_tool_gatewaybuild_vendor_gateway_url)tools.tool_backend_helpersmanaged_nous_tools_enabled)tools.url_safetynormalize_url_for_request)r0%nous_tool_gateway_unavailable_message)r0prefers_gateway)r.resolve_managed_tool_gateway)r2sensitive_query_param_name) r+r,r-r/r1r3r4r5r6r7ctj|}|tdtd|ddl}ddlm}|t|g|t|j|d|dS)Nzmodule z has no attribute r) warn_oncer) _PLUGIN_COMPAT_LAZYr0AttributeError__name__ importlibhermes_cli.plugin_compatr9r; import_module)rtargetr=r9s r% __getattr__rA1se $ $T *F ~wxl2DTHMNN2 h&v& 9**6!95vay AAr+rF)r)rNr()r)NN)Z__doc__rloggingr!typingrrr_firecrawl_client_firecrawl_client_config_parallel_client_async_parallel_client _exa_clientplugins.web.firecrawl.providerrrtools.debug_helpersrtools.tool_backend_helpersr r r tools.url_safetyr tools.web_tools_rescuer rtools.web_tools_truncaterrrrtools.web_tools_extractrrrrrrr getLoggerr<r<rr&r)r*rr1r6rBrGrKrPrkrgrtrwrer~r frozensetrfrr&rrrprrrrrrrr tools.registryr rWEB_SEARCH_SCHEMAWEB_EXTRACT_SCHEMAregisterr(r)asynciohttpxresysr:rArJr+r%r[s  &&immm,m/?mBX[fZ,^^.C||   8 $ IS IS I "3"4"$?S?? cNcN 0 S3"-c"-J(3-"CSC DcD $ +442i8.: .  !!67'3'4'*49 k+< =K(@S@4@HSM@]efi]j@-c3-cs-c3-c`7c7#7$7><ic<iC<iHUXM<ieh<i~^D^*UG$4D0 w! G "T   I. A   (+[ " d  H, u-> a ->-@!   .@ ->-@4W]!  #]@R Z"^!R-tH$b"T Br+