߲j, dZddlZddlZddlZddlmZmZmZmZddl m Z m Z ddl m Z ddlmZmZej"dZdZd Zd Zd Zd ed eefdZdeded efdZdeded efdZdeded efdZdedeed eeeffdZded efdZdefdZdedeee fdeedeedee d ee f d Z!d!eefd"Z"defd#Z#d e$fd$Z%deed%eed ee fd&Z&d'eed%eed ee fd(Z'y))aweb_extract helpers: URL validation, provider resolution, cache-aware dispatch. Order of controls (each is a gate, never skipped by a cache hit): secret-URL refusal -> SSRF filter (in web_tools.web_extract_tool) -> provider resolution (strict selection) -> per-URL website policy -> disk cache -> vendor call with one-shot keyless rescue. Logs under the origin (tools.web_tools) logger. N)AnyDictListOptional)selection_errorselection_exists)normalize_url_for_request)_rescue_eligible_rescue_extractztools.web_toolsz/Extract backend returned no result for this URLg^@z.firecrawl, tavily, keenable, exa, or parallel.zdInvalid URL item at index {}: expected a URL string or an object with a string 'url' or 'href' fieldvaluereturnct|tr$|jdxs|jd}t|tr|j xsdSdS)aURL from a model-supplied extract item (str, or dict with ``url``/``href``); None if unusable. Models sometimes forward a whole search result instead of its URL, hence the dict form. Never stringify arbitrary objects into misleading fetch targets. urlhrefN) isinstancedictgetstrstrip)r s Gs tTZZEE(JK KKrtotalfixedfetch_positions fetch_urlsresultsct|}t|D]0\}}t||t}|t |kr||n|||<2t |D cgc]} ||  c} Scc} w)zRebuild a ``total``-long result list: *fixed* entries by position, fetched *results* at *fetch_positions* (a short provider list yields ``_NO_RESULT_ERROR`` entries for the rest).)r enumerater3_NO_RESULT_ERRORlenrange) r?r@rArBrCmergedpospositionmissingis r_merge_in_orderrNLsq %[F"?3K X 31AB+.W+=73<7xK %U| ,!F1I ,, ,s A&urlsc v ddlm ddlm}ggi}}}t |D]\}}t |}|#t dtj|||<6t|}t fd||||||fDr tdcS|j||j||||dfS)aNormalize model-supplied items and block URLs carrying secrets (percent-encoded forms are unquoted and checked too). Returns ``(normalized_urls, normalized_indices, invalid_urls, blocked_json)``; ``blocked_json`` is a whole-call refusal (exfiltration prevention) or None.r) _PREFIX_RE)unquoteNr/c3@K|]}j|ywN)search).0crQs r z)_validate_extract_urls..fslz  #lsz_Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.) agent.redactrQ urllib.parserRrErr3_INVALID_ITEM_ERRORformatr anyr>append) rOrRnormalized_urlsnormalized_indices invalid_urlsindexitem_urlnormalized_urlrQs @r_validate_extract_urlsrfXs($8:B'O  ) t% <"/4G4N4Nu4U"VL  248 ldGDM>SZ[iSj-kl l4  ~.!!%( ) . d BBrc>ddlm}m}|r||nd}||jr|dfS|!dt |j dt zfS|r"tdrdt td|fS|}| dt z}dt td|fS|dfS)a#Resolve the extract provider for *backend*; returns ``(provider, error_json)``. A registered search-only backend is a typed error (never a silent switch). An unregistered name with a stored web selection is a strict-selection error; with no selection, fall through to the walk. r)get_active_extract_provider get_providerNzU is a search-only backend and cannot extract URL content. Set web.extract_backend to r)extractz?No web extract provider configured. Set web.extract_backend to ) r%rhrisupports_extractr< display_name_EXTRACT_BACKENDS_HINTrr,r&)r'rh_wsp_get_providerproviderr"s r_resolve_extract_providerrpps i-4 )$H 9 9 ;~($$%&* *,B C   #E*()@G)TUUU*,HTWmm();Ix)PQQQ T>rcddlm} t|jdtS#t t f$r tcYSwxYw)aWall-clock cap for one provider ``extract()`` dispatch (``web.extract_timeout``, default 120s). A hanging backend (server keeps the response open without finishing) otherwise stalls the tool call indefinitely. 0 or a negative value disables the cap. r_load_web_configextract_timeout)tools.web_toolsrsfloatr_DEFAULT_EXTRACT_TIMEOUT_S TypeError ValueErrorrrs r_extract_timeout_secondsrzsD 1*%'++,=?YZ[[ z "*))*s#,AAr\c Kddl}ddlm}t} |j |j r|j ||}n"t j|j ||}|dkDr t j||d{}n |d{}|rKt'd |Dr9t|r.t jt |j||d{St)||D]} | j+d } t-| t.r| j+d nd} t1fd | j+d| fDd}| j+ddxs| j+dd}|s|s| j+dr|||| j+dd||j|S707(#t j$r}tjd|j|t||D cgc]} t| d|dd |j!ncc} w} } t|s| cYd}~St jt |j|| d{7cYd}~Sd}~wt"$rm}t|s|D cgc]} t| t%|ncc} w} } t jt |j|| d{7cYd}~Sd}~wwxYw7w)uCall ``provider.extract`` (async or sync-in-thread), with one-shot keyless rescue. Rescue fires on a raised exception — including a dispatch timeout — or when the WHOLE batch failed (backend outage, not per-page problems). Rescued batches are never cached. rN)extract_cache_put)r\)timeoutz=web_extract provider '%s' timed out after %.0fs for %d URL(s)zExtract timed out after z.0fzs via c3>K|]}|jdyw)r-N)r)rVrs rrXz$_dispatch_extract..s7!quuW~7smetadata sourceURLc3,K|] }|vs| ywrTr2)rVu requesteds rrXz$_dispatch_extract..sN!qI~ANs r raw_contentr/r1r-r0r\ro)inspecttools.web_result_cacher|rziscoroutinefunctionrjasyncio to_threadwait_for TimeoutErrorloggerwarningnamerGr3r r Exceptionrallsetrrrnext)rorBr\rr|r}cororCexcrfailedfetchedmetasourcer_contentrs @r_dispatch_extractrs 8&(G[  & &x'7'7 8##Jv#>D$$X%5%5z&QD Q;#,,T7CCG jG37w77>?JX_ `` ! M !BCs@$DC6AD D=DD  D6 DDD D)(__doc__rr9loggingtypingrrrrtools.tool_backend_helpersrrtools.url_safetyr tools.web_tools_rescuer r getLoggerrrFrwrmr[rrr!r&r,r3r<r>intrrNrfrprvrzrrr2rrrs ,,H6D   , -D"Ij GCGHSMGs##Z3Z#Z#Z[[c[c[DsD8C=DT#s(^DNsNsNLsL -  -CI -9=c -PTUXPY -dhimdn - $Z -CcC0s0 *% */$s)/Xc]/W[\`Wa/da$s)aXc]aW[\`Waar