߲j׍UdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ejeZej"dZehdZdHd Zej"d ZdIdJd ZdHd ZdKd ZdLdZdMdZeeeZeeedZeeeZeeedZeZ dNdZ!dOdZ"dZ#dPdZ$dddZ%dQdZ&dRdSdZ'dTdUdZ(ddd d!Z)dVd"Z*dWd#Z+hd$Z,dXd%Z-d&Z.d'Z/dYd(Z0dZd)Z1dZd*Z2gd+Z3d[d,Z4d\d-Z5d]d^d.Z6d_d/Z7d`d0Z8dad1Z9dbd2Z:dcd3Z;d4ed5d6hed7d8fd9eed9hd:d;fd<eed=hd>d?ffZdddBZ?dedCZ@dfdDZAdgdEZBdhdFZCdidGZDycc}w)ja+Message and tool-payload sanitization helpers (pure; documented in-place mutation). Walk OpenAI-format message lists and structured payloads, repairing or stripping characters that would crash ``json.dumps`` in the OpenAI SDK or be rejected upstream. ``run_agent`` re-exports them for old imports. ) annotationsN)partial)AnyCallable)_provider_model_keyz[\ud800-\udfff]>namerolecontent tool_callsc.tjd|S)zHReplace lone surrogate code points with U+FFFD; no-op when none present.u�) _SURROGATE_REsubtexts ?/home/ubuntu/.hermes/hermes-agent/agent/message_sanitization.py_sanitize_surrogatesrs   Xt ,,z[A-Za-z0-9_-]{1,64}c t|ts|Stj|r|St j ddt j dd|j j d}|ddxs|S)uCoerce a *replayed* tool/function name to ``^[A-Za-z0-9_-]{1,64}$``. Valid names are returned as-is (identity — prompt-cache safe); invalid runs collapse to ``_`` and the result is cut at 64; empty/all-invalid → ``fallback``. Deterministic, so the same stored name always renders the same bytes. Never apply to live tool definitions (schema names must match the dispatch registry).z_+_z[^A-Za-z0-9_-]N@) isinstancestr_VALID_TOOL_NAME_RE fullmatchrerstrip)rfallbackcoerceds rcoerce_tool_namer(si dC $$T* ffUC(93 !MNTTUXYG 3B< #8#rcF|jddjdS)uSDrop non-ASCII characters — last resort for ASCII-only system encodings (LANG=C).asciiignoreerrors)encodedecoders r_strip_non_asciir'5s ;;wx; 0 7 7 @@rct|tr|j|n||}t|tr||n|}||k(ry|||<y)zGApply ``fix`` to ``container[key]`` if it is a str; True if it changed.FT)rdictgetr) containerkeyfixvaluefixeds r_fix_str_fieldr0:sI",Y"=IMM# 9S>E$UC0CJeE ~IcN rcld}|g}|r|j}t|tr|jnt|tr t |nd}t |D]N\}}t|t r|t|||z}'t|ttfs>|j|P|r|S)zHApply ``fix`` to every str inside nested dict/list ``payload`` in-place.F) poprr)itemslist enumeraterr0append)payloadr-foundstacknoder4r,r.s r_sanitize_structurer<Ds E IE yy{ *4 6 zZ^`dOeIdOkmu+ $JC%%c377ED$<0 U#  $  Lrcddlm}d}|D]}t|tsd}|j d}t|t r#|Dcgc]}t|ts|dfc}nd} | | n|dfg} | j |df|j d} t| t r| nd D]e} t| tr| j d nd} | |rt| tr| d fgngz } | t| tr|r| dfgng| d fgzngz } g| D]\}}|t|||z}|jDcgc]}|dtvs|c}D]O\}}t|tr|t|||z}'|s*t|tt fsA|t||z}Q|s|j|dd }|Scc}wcc}w)aApply ``fix`` to the string fields of every message dict in-place (content / part text, name, tool_call arguments, non-core top-level str fields). ``deep=True`` adds tool_call ids, function names, and NESTED non-core fields (``reasoning_details`` from byte-level models).r_DB_PERSISTED_MARKERFr rNrr r2functionid argumentsT) agent.context_compressorr?rr)r*r5r7r0r4_MESSAGE_CORE_KEYSrr<r3)messagesr-deepr?r9msg msg_foundr ppartsfieldsr tcfnr+r,kvr.s r_sanitize_messagesrOSs> E#t$  '')$IST[]aIbgEAt1D!VEhl+3 2B1C sFm$WW\* *:t <*" mB'1"d'; #B dz"d/CDzl KF PZ[]_cPdV ~22{:K9LLjl lF m% =NIs  3< >' " .:g3F'RWY`bfJg3g G"9*W:M"NJw %j1"rcXg}d}d}|t|kr||}|r3|dk(r.|dzt|kr|j|||dz|dz }H|dk(r| }|j|rt|dkrdt|d n||dz }|t|krd j|S) zEscape literal control chars (0x00-0x1F) inside JSON string values as ``\uXXXX`` (for llama.cpp-style output mixing control chars with other malformations).Fr\" z\u04x)lenr7ordjoin)rawout in_stringichs r%_escape_invalid_chars_in_json_stringsrksCI A c#h, V tAC(8 JJs1QU| $ FA  9% I )B$SR &BO Q c#h, 773<ricb tj|y#tj$rYywxYw)NTF)jsonloadsJSONDecodeErrorrs r _loads_okrps. 4   s ..}]){[cDg}g}d}dt|}}||kr||}|r*|dk(r|j|||dz|dz }.|dk(rd}n|dk(rd}n~|tvr|j|nd|dvr`|d|DvrUt|d |k7r6|jt|jt|d |k7r6|j|j||d z }||kr|ry d j |d j d t |DzS)uPClose a JSON prefix's open braces/brackets in stack order, ignoring delimiters inside string values (``{"code": "}"}`` keeps one open brace, not a balanced document). A closer that does not match the stack top but does match a deeper opener gets the missing inner closers inserted BEFORE it: ``{"a": [{"b": 1}}`` → the model dropped the ``]`` and let the neighbouring ``}`` close in its place, so the counts balance and nothing can be appended. ``None`` when the text ends inside an unterminated string — that content is unrecoverable and must not be guessed. Frr\r^r_Tz}]c3.K|] }t|ywN _JSON_CLOSERS).0os r z*_rebalance_json_closers..s"C=#3"Cr]Nrbc3.K|] }t|ywrwrx)rzrjs rr|z*_rebalance_json_closers..s!N-"3!Nr})rcr7ryr3rereversed)rfrgr:rhrinrjs r_rebalance_json_closersrs+CEI c#hqA a% V Tz 3qQ<(QSy! 3YI = LL  4ZB"CU"CCb *b0 =56 b *b0 IIK 2 Q% a%& 773<"''!Nhuo!NN NNrct|tr|jnd}|stj d|y|dk(rtj d|y t j t j|dd }||k7rtj d ||S#t jttf$rYnwxYwtjd d |}t|xs|}td D]|}t|sh|j!dr#|j#d|j#dkDs6|j!dr#|j#d|j#dkDsn|dd}~t|r tj d||dd|dd|St%|}||k7r+t|r tj d||dd|dd|Stj d||dt&y)zRepair malformed tool_call argument JSON (truncation, trailing commas, Python ``None``, control chars); ``"{}"`` if unrepairable so the request succeeds. Repairs log at WARNING.rbz*Sanitized empty tool_call arguments for %sz{}Nonez0Sanitized Python-None tool_call arguments for %sF)strict,:) separatorsz>Repaired unescaped control chars in tool_call arguments for %sz ,\s*([}\]])z\12rqrsrrrtNr~u8Repaired malformed tool_call arguments for %s: %s → %sPuARepaired control-char-laced tool_call arguments for %s: %s → %suPUnrepairable tool_call arguments for %s — replaced with empty object (was: %s))rrrloggerwarningrmdumpsrnro TypeError ValueErrorrrrrangerpendswithcountrk_FULL_ARGS_LOG_BOUND)raw_args tool_name raw_stripped reserialisedr/rescapeds r_repair_tool_call_argumentsrs(2(C'@8>>#bL CYOvI9U zz$**\%"HU_` < ' NN[]f g  )Z 8    FF>5, 7E #E * 3eE 2Y U  ^^C U[[%5 C8H%Hs# C(85;;s;K(K cr  QS\^jknln^oqvwzxzq{| 4E:G%Ig. OQZ\hiljl\movwzxzo{  NNZ< 5!56 sAB""CCc|r|dnd}t|tr|jddk7ryt|tr|nd}ddlm}||d |j xsd d y ) uWAppend a synthetic assistant turn when an interrupted tail is a tool result: a transcript ending on a raw ``tool`` message makes the next user message land as ``tool → user``, an alternation violation strict providers (Gemini, Claude) answer by hallucinating a continuation. Mutates in place; True if a closing turn was appended.r~Nr toolFrbr)append_message assistantzOperation interrupted.)r r T)rr)r*ragent.message_metadatarr)rEfinal_responselastrrs rclose_interrupted_tool_sequencers_ $8B<D dD !TXXf%5%?'<>"D58kdjjl>fNfgh rlengthstopr ) max_tokensend function_callcvt|tr|s|S|j}tj ||S)aFold a wire ``finish_reason`` to the lowercase OpenAI contract value. Non-string and empty values pass through unchanged (callers keep their ``or "stop"`` defaults and the Poolside int-reason path); contract values are returned byte-identical. )rrlower_FINISH_REASON_ALIASESr*)rflowereds rnormalize_finish_reasonr3s3 c3 s iikG ! % %gw 77rct|tr|sy ttj|ddt j dS#ttf$rtd|DcYSwxYw)zExact serialized byte size of ``messages`` (HTTP 413 is a BYTE-size error the token estimator, pricing images flat, cannot score). Non-serializable values fall back to ``str()`` so a malformed message can never crash recovery.rFr) ensure_asciirdefaultutf-8c3DK|]}tt|ywrw)rcr)rzms rr|z,serialized_messages_bytes..Is113s1v;1s ) rr5rcrmrrr%rrsum)rEs rserialized_messages_bytesr@sd h %X24::hUz[^_ffgnopp z "211112s4A !A.-A.>image image_url input_imagecfddlm}ddlm}d}g}t |D]\}}t |t r|jdnd}t |ts:|Dcgc],}t |t r|jdtvr+|.} }t| t|ksd}| r| |d<|j|dnN|jd d k(s|jd rd |d<|j|dn|j|||t|D]}||=|Scc}w) uRemove image content parts from all messages in-place (server rejected images). ``tool`` / ``tool_calls`` messages left empty get a placeholder, NOT deleted (deleting orphans the paired ``tool_call_id`` → HTTP 400); other now-empty messages are dropped. Rewritten messages lose their ``api_content`` sidecar (it carries the removed images): a caller rewriting a persisted row must not leave bytes that replay them next turn. The current callers pass per-call clones, where this is a no-op. rr>)drop_stale_api_contentFr NtypeTr rr u:[image content removed — server does not support images])rCr?agent.turn_contextrr6rr)r*r5_IMAGE_PART_TYPESrcr3r7r) rEr?rr9 to_deleterirGr rI new_partss r_strip_images_from_messagesrOs>9 EIH%(3(23(=#'')$4'4(  'l1 1d0Cf YjHjQl l y>CL (E!*I,d3F*cggl.C!]I,d3  # "3 '!("i  QK Lms ,D.D.)z%only 'text' content type is supportedz#only text content type is supportedzimage_url is not supportedzimage content is not supportedzmultimodal is not supportedz#multimodal content is not supportedz!multimodal input is not supportedzvision is not supportedzvision input is not supportedzdoes not support imageszdoes not support image inputzdoes not support multimodalzdoes not support visionzmodel does not support imagezunexpected item type in contentzimage_url'. expectedz,unknown variant `image_url`, expected `text`z(unknown variant image_url, expected textz+no endpoints found that support image input)z8image data you provided does not represent a valid imagezfailed to decode imageclt||jvryt|txr t |S)aSend-path image strip for a model that rejected image content (see turn_recovery). Runs on the per-call ``api_messages`` copy in Hermes's own message format, BEFORE the provider-specific conversion: the part types this stripper knows are that format's, and a converted payload (Bedrock Converse ``{"image": ...}`` blocks carry no ``type``) would slip past it. History is never touched. Keyed on each rejecting (provider, model), so a model that accepts images gets them again. F)r_image_rejecting_modelsrr5r)rX api_messagess r strip_images_for_rejecting_modelrs25!)F)FF lD ) W.I,.WWrcnt|xsdjtfdtDS)zMReturn True when a provider error says image/multimodal input is unsupported.rbc3&K|]}|v ywrwr2rzphrasebodys rr|z6_looks_like_image_content_rejection..sE&v~E)rrany_IMAGE_REJECTION_PHRASES error_bodyrs @r#_looks_like_image_content_rejectionrs- zR & & (D E,DE EErcnt|xsdjtfdtDS)zXReturn True when the rejection is about a bad image payload, not the model's capability.rbc3&K|]}|v ywrwr2rs rr|z6_looks_like_corrupt_image_rejection..sC&v~Cr)rrr_IMAGE_CORRUPT_PHRASESrs @r#_looks_like_corrupt_image_rejectionrs- zR & & (D C,BC CCr)r rrrS_sanitize_messages_surrogatesrrkrr'_sanitize_messages_non_ascii_sanitize_tools_non_asciirrWrZrdeterministic_call_idcoalesce_tool_call_idtool_call_id_variantstool_result_id_variantsuniquify_tool_call_idsreasoning_echo_familymatches_reasoning_echo_familyneeds_reasoning_echostale_thinking_reaches_wireapply_reasoning_content_policyreapply_reasoning_echoc^t|tr|j|St||dS)zHRead ``key`` from a tool-call entry that may be a dict or an SDK object.N)rr)r*rU)rLr,s r _tc_fieldrs'$R.266#;JGBT4JJrcdt|tr|j||yt|||yrw)rr) __setitem__setattr)rLr,r.s r_tc_setrs$",R"6BNN3GBU>$++gi+"HISSUVYWYZ[ \\rct}|D]f}t|tr|jnd}|s(|j ||j dd|j dDDht|S)zEvery wire spelling of one tool-call identifier: Responses bridges may expose the pairing id and response-item id separately or as ``call_id|response_item_id``; all alias ONE call.rbc3&K|] }|s| ywrwr2)rzrIs rr|z+_expand_tool_id_variants..sX!VWAXsc3<K|]}|jywrw)r)rzparts rr|z+_expand_tool_id_variants..s'R 'Rs|)setrrraddupdatesplit frozenset)valuesvariantsrfr.s r_expand_tool_id_variantsrsoHY)#s3   LL  OOX'RSAQ'RX X Y X rc>ttfddDS)z.s)jq)B*:)js)call_idrAresponse_item_id)rtuple)rLs`rrrs #E)jDi)j$j kkrct|fS)z>Return all matching variants for a role=tool ``tool_call_id``.)r) tool_call_ids rrrs #\O 44rct|dt|dfD]O}t|tr|jnd}|s(|j dddjxs|cSy)zEffective call id of a tool_call entry (dict or object); ``""`` when none. Codex Responses carry ``call_id`` (authoritative pairing key), Chat Completions ``id`` only, and bridge ids may be ``call_id|response_item_id``.rrArbrr]r)rrrrr)rLrfr.s rrrsh"i()B*=>;)#s3  ;;sA&q)//1:U :; rc t|xsgD]@}t|dxst|dxsd}t|tr|j nd}|j ddds\vrj rtfdtdtd zD}j | t|d}t|dt|trd|vr|d|j dddn|t|dr t|d|tt|d d xsd }tjd||C|S#t$rtjd YhwxYw)uEnsure every tool call in one assistant turn has a distinct id. Some providers reuse one id across a batch; the pre-API sanitizer then keeps only the first call/result pair per id and strict providers reject duplicates. Later collisions get a deterministic ``_d`` suffix (never uuid4 — cache-prefix stability). Mutates entries (SDK models / SimpleNamespace / dicts) in place. Blank ids are left for the deterministic fallback in ``build_assistant_message``. rrArbrr]rc3@K|]}d|vs d|yw)_dNr2)rzrcidseens rr|z)uniquify_tool_call_ids..!s/b3%rRSQT ]aHaRsmbs r^z,Could not uniquify duplicate tool call id %sr@r?zyModel reused tool call id %s within one turn; renamed the duplicate to %s (tool=%s) to keep call/result pairing lossless.)rrrrrrrnextrrcr Exceptionrr)r rLrfnew_idold_fn_namer r s @@rrr spDB I&C)B*=C'S1ciikriiQ"  d? HHSM bU1c$i!m-Dbb  B%C BJsTWDX]`dg]g#))C*;A*>)?@ms tY'Iv.Yr:6?F3 ,-0&( 3 :   NNI3 O  sA EE<;E<kimiz kimi-codingzkimi-coding-cnr2)z api.kimi.comz moonshot.aiz moonshot.cndeepseek)r)zapi.deepseek.commimoxiaomi)r)zapi.xiaomimimo.comzxiaomimimo.comr_REASONING_ECHO_RULESc ddlm t|\}}}}}|xsdj ||vxsC|xsdj|vxs+t fd|Dxst  fd|DS)zTrue when (provider, model, base_url) matches one echo-back family (families can overlap; membership is tested independently). Raises KeyError for an unknown family.r)base_url_host_matchesrbc3&K|]}|v ywrwr2)rzr model_lowers rr|z0matches_reasoning_echo_family..Zs8csk!8rc30K|] }|ywrwr2)rzhostbase_urlrs rr|z0matches_reasoning_echo_family..Zs?xjn@UV^`d@e?xs)utilsr_REASONING_ECHO_RULE_BY_FAMILYrr) familyprovidermodelrr raw_providerslowered_providers model_subshostsrrs ` @@rrrQs,=[\b=c:A}'U;B%%'KM! yhn"%;%;%=AR%R y 8Z8 8 y.as:DQ:sc3@K|]}t|s|ywrw)r)rzfrr#r"s rr|z(reasoning_echo_family..bs!dq(EaSXZb(cdsN)rr)r"r#rfamiliess``` rrr^s$;$9:H dHdfj kkrc t|||duS)zr<r?)rEr5r-r>rFr?r<r?)rXrrYr)r<r)rfrr<r)rrr<r?)rfrr<z str | None)r )rrrrr<rrw)rEr5rrr<r?)rfrr<r)rEr5r<int)rEr5r<r?)rXrrrr<r?)rrr<r?)rLrr,rr<r)rLrr,rr.rr<r)r)rrrBrrr@r<r)rztuple[Any, ...]r<frozenset[str])rLrr<rA)rrr<rA)rLrr<r)r r5r<r5) r!rr"rr#rrrr<r?)r"rr#rrrr<z 'str | None')r"rr#rrrr<r?) r1rr"rr#rrrr<r?)r6r)r7r)r8r?r<r)rr5r8r?r<r@)E__doc__ __future__rrrmloggingr functoolsrtypingrragent.vision_message_prepr getLogger__name__rcompiler rrDrrrr'r0r<rOrSrrWrrrZrkrrpryrrrrrrrrrrrrr__all__rrrrrrrrr__annotations__r rrrrrr)r*s0rrMs-#   9   8 $ -. HI-!bjj!78 $A   J"))