+ lHjw,6Rt^RIt^RIt^RIt^RIHtHtHtHt^RI H t H t ^RI H t ^RIHtHt]P"!R4tRtRtR tR tR R ltR RltRRltRRltRRltRRltRRltRRltRRltRRlt RR lt!R!R"lt"R#R$lt#R#)%aweb_extract helpers: URL validation, provider resolution, cache-aware dispatch. Order of controls (each is a gate, never skipped by a cache hit): secret-URL refusal -> SSRF filter (in web_tools.web_extract_tool) -> provider resolution (strict selection) -> per-URL website policy -> disk cache -> vendor call with one-shot keyless rescue. Logs under the origin (tools.web_tools) logger. N)AnyDictListOptional)selection_errorselection_exists)normalize_url_for_request)_rescue_eligible_rescue_extractztools.web_toolsz/Extract backend returned no result for this URLg^@z.firecrawl, tavily, keenable, exa, or parallel.zdInvalid URL item at index {}: expected a URL string or an object with a string 'url' or 'href' fieldcFV^8dQhR\R\\,/#)valuereturn)rrstr)formats"RRR\P!RRRV/43#)zRWhole-call refusal tuple for ``_validate_extract_urls`` (exfiltration prevention).Nr=Fr4r?rBs&r _refuse_allrFGs# tTZZE7E(JK KKrc V^8dQhR\R\\\3,R\\,R\\,R\\,R\\,/#)r totalfixedfetch_positions fetch_urlsresultsr)intrlistr)rs"rrrLsX - -  -CI -9=c -PTUXPY -dhimdn - $Z -rc\V4p\V4F7wrg\W6,\4pV\ V48d WF,MTWW&K9 \ V4U u.uF qV ,NK up #uup i)zRebuild a ``total``-long result list: *fixed* entries by position, fetched *results* at *fetch_positions* (a short provider list yields ``_NO_RESULT_ERROR`` entries for the rest).)r enumerater:_NO_RESULT_ERRORlenrange) rHrIrJrKrLmergedpospositionmissingis &&&&& r_merge_in_orderrYLsf %[F"?3  1AB+.W+=7<74 %U| ,|!1II| ,, ,sA3c:V^8dQhR\\,/#)r urls)rNr)rs"rrrXsCCcCrc a ^RIHo ^RIHp../rCp\ V4FwrV\ V4pVf$\ R\PV44WE&K7\V4p\;QJd-V 3RlWq!V4W!V434F 'gK RM" RM!V 3RlWq!V4W!V4344'd\R4u#VPV4VPV4K W#VR3#) aNormalize model-supplied items and block URLs carrying secrets (percent-encoded forms are unquoted and checked too). Returns ``(normalized_urls, normalized_indices, invalid_urls, blocked_json)``; ``blocked_json`` is a whole-call refusal (exfiltration prevention) or None.) _PREFIX_RE)unquoteNr7c3F<"TFpSPV4xK R#5iN)search).0cr]s& r )_validate_extract_urls..fs!l-kz  ##-ks!TFz_Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.) agent.redactr] urllib.parser^rPrr:_INVALID_ITEM_ERRORrranyrFappend) r[r^normalized_urlsnormalized_indices invalid_urlsindexitem_urlnormalized_urlr]s & @r_validate_extract_urlsrrXs($8:BO  % <"/4G4N4Nu4U"VL  248 3ldGDM>SZ[iSj-kl333ldGDM>SZ[iSj-kl l l4  ~.!!%('  d BBrc$V^8dQhR\/#)r r-r)rs"rrrpssrc^RIHpHpV'd V!V4MRpVeVP4'dVR3#Ve&R\ VP R2\ ,43#V'd)\R4'dR\ \RV443#V!4pVf%R\ ,pR\ \RV443#VR3#)aResolve the extract provider for *backend*; returns ``(provider, error_json)``. A registered search-only backend is a typed error (never a silent switch). An unregistered name with a stored web selection is a strict-selection error; with no selection, fall through to the walk. )get_active_extract_provider get_providerNzU is a search-only backend and cannot extract URL content. Set web.extract_backend to r/extractz?No web extract provider configured. Set web.extract_backend to ) r*rurvsupports_extractrC display_name_EXTRACT_BACKENDS_HINTrr2r+)r-ru_wsp_get_providerproviderr's& r_resolve_extract_providerr}ps i-4 )$H 9 9 ; ;~($$%&* *,B C   #E**()@G)TUUU*,HTWmm();Ix)PQQQ T>rc$V^8dQhR\/#)r r)float)rs"rrrs * *% *rc^RIHp\V!4PR\44# \ \ 3d \u#i;i)zWall-clock cap for one provider ``extract()`` dispatch (``web.extract_timeout``, default 120s). A hanging backend (server keeps the response open without finishing) otherwise stalls the tool call indefinitely. 0 or a negative value disables the cap. _load_web_configextract_timeout)tools.web_toolsrrr_DEFAULT_EXTRACT_TIMEOUT_S TypeError ValueErrorrs r_extract_timeout_secondsrsD 1*%'++,=?YZ[[ z "*))*s#,AAc~V^8dQhR\\,R\\,R\\,/#)r rKrrrNrrr)rs"rrrs0//$s)/Xc]/W[\`Wa/rc a"^RIp^RIHp\4pVP VP 4'dVP WR7pM"\ P!VP WR7pV^8d!\ P!WeR7GRjxL pM VGRjxL pV'dz\&;QJdR V4F 'dK R M R M !R V44'd@\V4'd/\ P!\ VPW4GRjxL #\)V4oVFp V P+R 4p \-V \.4'dV P+R 4MRp \1V3RlV P+R4V 34R4pV P+RR4;'gV P+RR4pV'gKV'gKV P+R4'dKV!WV P+RR4W PR7K V#ELEL \d\PRTPT\T44TUu.uF p\TRTR RTP 24NK" Muupip p\T4'gT u#\ P!\ TPY4GRjxL u#\"dvp \T4'ghTUu.uFp\T\%T 44NK Muupip p\ P!\ TPY4GRjxL uRp ? #Rp ? ii;iEL05i)uCall ``provider.extract`` (async or sync-in-thread), with one-shot keyless rescue. Rescue fires on a raised exception — including a dispatch timeout — or when the WHOLE batch failed (backend outage, not per-page problems). Rescued batches are never cached. N)extract_cache_put)r)timeoutz=web_extract provider '%s' timed out after %.0fs for %d URL(s)zExtract timed out after z.0fzs via c3B"TFqPR4xK R#5i)r4N)r)rbrs& rrd$_dispatch_extract..s7w!uuW~~wsFTmetadata sourceURLc38<"TFqS9gK VxK R#5ir`r9)rbu requesteds& rrdrsN;!I~AA;s r raw_contentr7r8r4r6rr|)inspecttools.web_result_cacherriscoroutinefunctionrwasyncio to_threadwait_for TimeoutErrorloggerwarningnamerRr:r r Exceptionrallsetrrrnext)r|rKrrrrcororLrfailedexcfetchedmetasourcer_contentrs&&& @r_dispatch_extractrs 8&(G[  & &x'7'7 8 8##J#>D$$X%5%5zQD Q;#,,TCCG jG337w73337w777=M M)M KM  M M M M+L  1M:L=;M?M MM  Mc~V^8dQhR\\,R\\,R\\,/#)r safe_urlsrrr)rs"rrrs5aa$s)aXc]aW[\`WaarcF"^RIHp^RIHp/..rvp\ V4FTwrV!V 4p V fV!WVP R7MRp V eWV&K2VPV 4VPV4KV V'g+\\V44U u.uF qV ,NK up #\PRVP \V44\WV4GRjxL p V'gV #\\V4WWWm4# \ dRp Li;iuup iL<5i)uServe cache hits, fetch the rest, and merge back in ``safe_urls`` order. The disk cache (tools/web_result_cache.py) sits AFTER the secret-URL gate, SSRF gate, and provider resolution, and is gated per-URL on the website policy — a hit skips only the vendor call, never a control; policy-blocked URLs are cache misses. Keys include provider and format, so switching either within the TTL never serves the other's content.)extract_cache_get)check_website_accessNrzWeb extract via %s: %d URL(s))rrtools.website_policyrrPrrrjrSrRrinforrY)r|rrr _check_sitecached_resultsrKrJrVr _policy_blockhitrXrLs&&& r_extract_safe_urlsrs 9H24b"N"9-  !',MP]OdX]]Kjn ?'*8 $   c "  " "8 ,. +0Y+@A+@aq!!+@AA KK/JP%hFCCG  3y>>J `` ! M !BCsL"D!DAD!?D!D&rs ,,H6D   , -D"Ij GZ[DNL -C00 */dar