߲jq dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZej$eZGddZej,Zd e ej0d dfd Zd e ee fd ede ded df dZej:dej<dfej:dej<dfej:dej<dfej:dej<dfej:ddfej:ddfej:dej>dfej:ddfej:ddff Z ded efdZ!Gdd Z"d!ed efd"Z#ej:d#Z$d$ed e%fd%Z&d$ed e'efd&Z(d'e'ed efd(Z)ded efd)Z*ddddd*d+ed,edzd-edzd.d/d0d1d d2f d3Z+ded e%fd4Z,ded e%fd5Z-ded e%fd6Z.d$ed e%fd7Z/ded e%fd8Z0ej:d9Z1d:Z2dUd;Z3ded ded?ed efd@Z5dVdAZ6dVdBZ7dWdedDe%dEed dFfdGZ8dXdHZ9dUdIdCdJdKZ:dUdLZ;dMZd e%fdPZ?ddQl m@Z@ddRl mAZAddlZddlZGdSdTZBy)YuShared helpers for gateway platform adapters: message dedup, markdown stripping, thread participation tracking, GFM table → bullets, mention-pattern compilation, and fence-aware markdown chunking.N)Path)AnyMutableMappingOptional) MessageEvent)atomic_json_writecXeZdZdZd dedefdZdedefdZ dedefdZ dedd fd Z d Z y ) MessageDeduplicatorzRTTL-based message deduplication cache (``if dedup.is_duplicate(msg_id): return``).max_size ttl_secondsc.i|_||_||_yN)_seen _max_size_ttl)selfr r s >/home/ubuntu/.hermes/hermes-agent/gateway/platforms/helpers.py__init__zMessageDeduplicator.__init__s') ! msg_idreturncn|sytj}||jvr-||j|z |jkry|j|=||j|<t|j|jkDr||jz }|jj Dcic]\}}||kDs ||c}}|_t|j|jkDrBt t|jj d|j d|_ycc}}w)z?Return True if *msg_id* was already seen within the TTL window.FTc |dS)N)items rz2MessageDeduplicator.is_duplicate..+s dSTgr)keyN)timerrlenritemsdictsorted)rrnowcutoffkvs r is_duplicatez MessageDeduplicator.is_duplicatesiik TZZ TZZ''$))3 6"  6 tzz?T^^ +499_F+/::+;+;+=L41aV!Q$LDJ4::/!&)9)9);AU"VX\XfXfWfWg"hi  Ms 0 D1>D1c|r|jj|nd}|ytj|z |jkry|j|=y)zBReturn whether *msg_id* is live in the cache without inserting it.NFT)rgetrr)rrseen_ats rcontainszMessageDeduplicator.contains.sI,2$**..( ? 99; 499 , JJv rNc<|jj|dy)z_threads.json``); ``thread_id in tracker`` checks membership, ``tracker.mark(thread_id)`` persists. platform_name max_trackedc||_||_tj|_tj|_t jd|jD|_ y)Nc32K|]}t|ywr)r9).0ts r z6ThreadParticipationTracker.__init__..s6T!s1v6Ts) _platform _max_tracked threadingLock_lock_io_lockr"fromkeys_load_threads)rrZr[s rrz#ThreadParticipationTracker.__init__wsK&'^^% !( )-6Ttzz|6T)T rrc<ddlm}||jdz S)Nr)get_hermes_homez _threads.json)hermes_constantsrkra)rrks r _state_pathz&ThreadParticipationTracker._state_paths4 dnn%5]#CCCrc tj|jjd}t |t r|Dcgc] }t|c}SgS#t$rgcYSwxYwcc}w)Nzutf-8)encoding)jsonloadsrm read_textrD isinstancelistr9)rdata thread_ids rrhz ThreadParticipationTracker._loadsk ::d..0::G:LMD:DD$9O59I5WUWW I 5s3A A1 A.-A.Ncn|j5|j5t|j}t ||j kDr*||j d}t j||_dddt|jddddy#1swY.xYw#1swYyxYw)N)indent) rfrertrir rbr"rgrrm)r thread_lists r_savez ThreadParticipationTracker._saves ]] L ?"4==1 {#d&7&77"-t/@/@.@.A"BK$(MM+$>DM  ? d..0+d K  L L ? ? L Ls# B+AB2$B+B( $B++B4rvc|j5||jvr dddyd|j|< dddy#1swYyxYw)a2Record *thread_id* in memory; ``True`` when a persist is still owed. The in-memory half stays synchronous even for :meth:`mark_async` so that a ``thread_id in tracker`` check immediately after marking is correct -- the mention-gating logic in the adapters depends on that. NFTrerirrvs r _rememberz$ThreadParticipationTracker._remembersLZZ DMM)  (,DMM) $    s ??AcH|j|r|jyy)zMark *thread_id* as participated and persist. Blocking: ends in ``atomic_json_write`` -> ``os.replace``. Coroutines must use :meth:`mark_async` instead. N)r~rzr}s rmarkzThreadParticipationTracker.marks >>) $ JJL %rcnK|j|rt|jd{yy7w)aOff-loop form of :meth:`mark`. ``_save`` ends in ``atomic_json_write`` -> ``os.replace``, whose duration is unbounded under filesystem pressure. Every caller of this tracker sits on an inbound-message coroutine in the Discord and Matrix adapters, so the rename must not be paid inline on the event loop -- it stalls every other adapter's polling and every in-flight turn for as long as it runs. N)r~ _to_threadrzr}s r mark_asyncz%ThreadParticipationTracker.mark_asyncs. >>) $TZZ( ( ( % (s *535cb|j5||jvcdddS#1swYyxYwrr|r}s r __contains__z'ThreadParticipationTracker.__contains__s* ZZ . - . . .s%.cz|j5|jjdddy#1swYyxYwr)rerir1r2s rr1z ThreadParticipationTracker.clears. ZZ " MM   ! " " "s1:)rYrN)r3r4r5r6 _MAX_TRACKEDr9r7rrrmrtrhrzr:r~rrrr1rrrrXrXqsYLUcUU$DTDXtCyXL 3 4 cd )# )$ ).c.d."rrXphonec||syt|dkrt|dkDr|dddz|ddzSdS|dddz|ddzS) zFRedact a phone number for logging, preserving country code and last 4.zNz****)r )rs r redact_phonersZ  5zQ25e*q.uRay6!E"#J.LfL !9v bc **rz0^\s*\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*){1,}\|?\s*$linec&d|jvS)z:Return True if *line* could plausibly be a table data row.|)rS)rs r is_table_rowrs $**, rcddlm}||S)zOSplit a GFM table row into stripped cells (delegates to agent.markdown_tables).r)split_table_row)agent.markdown_tablesr)rrs rsplit_markdown_table_rowrs5 4  r table_blockc t|dk\rt|dng}t|dkrdj|Stt|dt|dzk(}g}t|dddD]\}}t|}|r|r |dr|dnd|}|dd}nt d |Dd|}|}|d gt|zzdt|}t ||D cgc]\} } |s| |k7rd | d | } } } |j djd |d g| dj|Scc} } w)zRender a GFM table as bold-heading + bullet groups (same alignment logic as Telegram's renderer: without a row-label column the full row is data and the heading bullet is skipped).rr rN)startzRow c3&K|] }|s| ywrr)r^cells rr`z&_render_table_block..s;TdD;srMu• z: z**rN)r rjoin enumeraterJzipappend) rheadershas_row_label_colrendered_groupsindexrowcellsheading data_cellsheaderrGbulletss r_render_table_blockrsr;>k:Ja:O&{1~6UWG 7|ayy%%4[^DEWXYIYY!#O ABq9 H s(- "'E!HeAhD.GqrJ;U;tE7^LGJ B4#g,#66#g,G ADWjAY= '5G+;&E7+==tyyBwir*:)EW)EFG H ;; ''=s%D;cLd|vsd|vr|S|jd}g}d}d}|t|kr||}|jjd}||z}|s|sd|vr|dzt|kr~tj ||dzrc|dz}|t|kr0t ||r"|dz }|t|krt ||r"|jt||||}n|j||dz }|t|krdj|S) zURewrite GFM pipe tables into bold-heading + bullet groups; fenced code is left alone.r-rFr```rr) splitr lstrip startswithTABLE_SEPARATOR_REmatchrrrr)rOlinesoutin_fenceir is_fence_linejs rconvert_table_to_bulletsrs$ $#T/ JJt ECH A c%j.Qx 007 M!Msd{q1us5z?Q&,,U1q5\:AAc%j.\%(%;Qc%j.\%(%; JJ*51:6 7A JJt  FA c%j. 99S>r)platform_label display_labeldefaultslogger_ log_prefixrrrzlist[str] | Nonerzlogging.Logger | Nonezlist[re.Pattern]c |xstfd}||xs|}|gSt|tr|gn|}t|ts)j d|t |j gS||D cgc]&} t| ts| js%| (c} d|} | rjdt| || S|t|xsg}nt|tr|j} | rtj| ng} t| tr| nC| jD cgc](} | jdD]}|j*c}} }nt|tr|n|g}d|DDcgc]}|s| }}||dScc} w#t$rd} YwxYwcc}} wcc}w) a9Compile regex wake-word/mention patterns from config or env values. * **Config-style** (dingtalk, telegram): pass ``platform_label``. ``raw`` must be a list or string (else warn + ``[]``); non-string entries skipped; info log on load. * **Wakeword-style** (photon, bluebubbles): pass ``defaults``. ``raw`` may be None (defaults), a string (JSON list or comma/newline separated), a list, or a scalar. ``log_prefix`` is interpolated into every log line so per-adapter output stays byte-identical to the historical inline implementations. cg}|D]6} |jtj|tj8|S#tj$r"}j |g|||Yd}~kd}~wwxYwr)rrecompile IGNORECASEerrorwarning)patternswarn_fmt warn_argscompiledrTexclogrs r_compilez*compile_mention_patterns.._compile"s{%' LG L 7BMM BC L 88 L HjK9KgKsKK Ls3AA6A11A6Nz9[%s] %s mention_patterns must be a list or string; got %sz&[%s] Invalid %s mention pattern %r: %sz$[%s] Loaded %d %s mention pattern(s),c3NK|]}t|jywr)r9rS)r^ps rr`z+compile_mention_patterns..Es:AQ:s#%z#[%s] Invalid mention pattern %r: %s)loggerrsr9rtrtyper3rSinfor rprqrD splitlinesr)rawrrrrrrdisplayrrrrOloadedrpartr_textsrs ` @rcompile_mention_patternsrs  VC!1> ;I&sC0C5c(D) KKS"NDN4K4K MIU1Jq#4F1779QUDgO  HH;ZXX_ ` {B' C yy{ )-TZZ%2F(56%)__%6F>' F54F5cPtd|jdDdzdk(S)z?Return True when *text* ends inside an unclosed ``` code fence.c3>K|]}|jdyw)rN)r)r^rs rr`z*text_has_unclosed_fence..PsC$tu%Csrrr)sumrrOs rtext_has_unclosed_fencerNs& C$**T2BC Ca G1 LLrcXt|jjddS)z=True when the last non-empty line starts and ends with ``|``.r) _is_pipe_rowrstriprrs rtext_ends_with_table_rowrSs#  ++D1"5 66rc@|jjdS)z(9(9#(>>rc<t|jddS)z?True when an atomic block is a table (first line is ``|...|``).rr)rrrs r is_table_atomrbs  4(+ ,,ru[。！？.!?]\nc6|tur|Sddlm}||||S)zCode-point count of the longest prefix of *text* within *budget* ``len_fn`` units (callers guarantee ``len_fn(text) > budget``, so plain ``len`` needs no search).r)_custom_unit_to_cp)r gateway.platforms.baser)rObudgetlen_fnrs r _cp_budgetrjs"} 9 dFF 33rcj|xst}|||kr|dfS|dt|||}|jd}tj |Dcgc]}|j }}|dkDr|dzn|r|dnd}|s&|jd}|dkDr|dzn t|}|d|||dfScc}w) uMSplit at the nearest paragraph boundary within *max_chars*; return (head, tail). Priority: blank line → newline after sentence-ending punctuation (CJK/ASCII) → last newline → forced split at the window boundary. ``head + tail == text`` always holds. *len_fn* measures in custom units (e.g. UTF-16 code units). rMNrNrrrrr)r rrfind_SENTENCE_END_NEWLINE_REfinditerend) rO max_charsr_lenwindowposm sentence_endscuts rsplit_at_paragraph_boundaryrss =SD DzYRx 4:dIt4 5F ,,v C&>&G&G&OPQUUWPMP1W#' =#41C ll4 qcAgc&k :tCDz !! QsB0 list[str]cggd}dfd }|jdD]}|r-j||jds(d}|2|jdr|d}j|^|jdk(r|yr!t dt |k7r|j||S) zSplit markdown into indivisible atoms: fenced code blocks, tables (consecutive ``|...|`` lines) and paragraphs. Blank lines belong to no atom.FcrCdj}|jrj|jyy)Nr)rrSrr1)atomatoms current_liness r_flush_currentz,split_markdown_atoms.._flush_currents: 99]+Dzz| T"    ! rrrTrMrr)rrrrSr)rOrrrrrs @@rsplit_markdown_atomsr sE!#MH" 4 '    &u%  __U #  H   & ZZ\R   mB.?!@LQUDV!V   &!'" Lr prev_chunk next_chunkc|j}|j}|jds|jdryt |r |rt |j ddryy)zV``'\n'`` when the boundary sits at a code fence or a continued table, else ``'\n\n'``.rrrrN)rrrrrrr)r r  prev_trimmed next_trimmeds rinfer_block_separatorrse$$&L$$&LU#|'>'>u'E + lN`N`aeNfghNiAj rc6g}d}|t|kr||}t|rQ|dzt|kr@|t|||dzz||dzz}|dz }t|r|dzt|kr@|j||dz }|t|kr|S)zRejoin chunks truncated mid-fence: while chunk *i* has an unclosed fence and a successor exists, merge the successor in via :func:`infer_block_separator`.rr)r rrr)chunksresultrcurrents rmerge_streaming_fencesrsF A c&k/)%g.1q53v;3F 5gva!e} MMPVWX[\W\P]]G FA&g.1q53v;3F  g Q c&k/ Mrct|dkr|Sg}d}|D]B}|d|d|n|}t||du|xsd\}}|r|nd}|j|r|dzn|D|S)zClose orphaned ``` fences at each chunk boundary and reopen (with the original language tag) on the next, so every chunk is fence-balanced alone.rNrrrMz ```)r fence_state_afterr)rr carry_langchunkbodyin_codelangs rbalance_fences_across_chunksrs 6{a CJ8.8.DZL5'*%)%41GIYWYZ $T$  W4'>$7 8 JrFrrztuple[bool, str]c|jdD]I}|j}|jds%|ddj}|rdn d|r|dndf\}}K||fS) zSWalk ``text`` line by line toggling on ``` lines; return the final (in_code, lang).rrrNFrMTrrM)rrSr)rOrrrrtags rrrsp 4 V::<   u %12,$$&C+2Ks1vRT8UMGT V D=rc&|xst}g}d}|D]j}|s|n|||} || |kr| }|r|j|d}|||kr|}@||j||Z|j|l|r|j||S)zGreedily pack *blocks* (joined with *sep*) into chunks of at most *max_length*; an oversized block goes through *overflow(block)* (-> list of chunks) if given, else as-is.rM)r rextend) blocks max_lengthrsepoverflowrpackedrblock candidates rgreedy_pack_blocksr)s =SDFG !!(E #ug.F  ?j (G   MM' "G ;* $G  ! MM(5/ * MM%  ! g MrT)prefer_paragraphsbalance_fencescr|sgS|rt|||}nt|||xst}|r t|S|S)aESplit markdown into chunks of at most *limit*, respecting fences. ``prefer_paragraphs=True`` (yuanbao-derived): atoms (fences, tables, paragraphs) are greedily merged, oversized non-atomic chunks split at paragraph boundaries, small neighbours re-merged; a single atom larger than *limit* is emitted oversize, not broken. ``prefer_paragraphs=False`` (stream_consumer-derived): newline-preferred hard splitting with headroom reserved for fence markers. ``balance_fences=True`` closes/reopens fences at chunk boundaries so each chunk renders standalone. )_chunk_markdown_paragraphs_chunk_newline_preferredr r)rOlimitrr*r+rs rsplit_text_fence_awarer0sB  +D%@)$v}E3A ' /MvMrc|xst}|||kr|gSg}t}g}d}t|D]}||} |rdnd} || z| z|kDr(|r&|jdj |gdd} }}|sG| |kDrBt |s t |r,|jt||j||j||| | zz }|r |jdj |g} t|D]\} } || |ks| |vs t| r| j| 2| }|||kDrY &!F2J MM% !  #!A ## #s G+$G+cT|||kr|gSd|vrt|dz |dzdn|}g}|}|||kDr^t|||}|jdd|}||dzkr|}|j|d|||dj d}|||kDr^|r|j||S)zCStream-consumer-derived newline-preferred splitting (no balancing).rrrrrN)maxrrrr)rOr/r split_limitrr:rsplit_ats rr.r.=s d|uv 49TM#ebj%1*a0uKFI  k )I{F;??4F3 fk !H i *+hi(//5  k ) i Mri ___hidden___c |jj}t|tr |t zryt |ddtk(S#t$rd}Y;wxYw)uTrue when a discord.py channel object is an obfuscated placeholder. Skip these wherever guild channels are enumerated (channel directory, message backfill): history reads and sends always fail with a permission error. Flag first; the sentinel name is the fallback for discord.py builds that don't expose the flag (a visible channel literally named ``___hidden___`` is then also skipped — deliberate bias toward hiding). #90154 NTname)flagsrGAttributeErrorrsr7DISCORD_CHANNEL_OBFUSCATED_FLAGgetattrDISCORD_OBFUSCATED_CHANNEL_NAME)channel flag_values ris_discord_channel_obfuscatedrNYsX]](( *c"z4S'S 7FD )-L LL  sA AA)Dict) TYPE_CHECKINGcfeZdZdZdddddededefd Zd efd Zd e d e d dfdZ d e d dfdZ ddZ y)TextBatchAggregatora@Aggregates rapid-fire text events into single messages. Replaces the ``_enqueue_text_event`` / ``_flush_text_batch`` pattern previously duplicated in telegram, discord, matrix, wecom, and feishu. Usage:: self._text_batcher = TextBatchAggregator( handler=self._message_handler, batch_delay=0.6, split_threshold=1900, ) # In message dispatch: if msg_type == MessageType.TEXT and self._text_batcher.is_enabled(): self._text_batcher.enqueue(event, session_key) return g333333?g@i) batch_delay split_delaysplit_thresholdrSrTrUcX||_||_||_||_i|_i|_yr)_handler _batch_delay _split_delay_split_threshold_pending_pending_tasks)rhandlerrSrTrUs rrzTextBatchAggregator.__init__s2  '' /13 79rrc |jdkDS)z.Return True if batching is active (delay > 0).r)rXr2s r is_enabledzTextBatchAggregator.is_enableds  1$$reventrNct|jxsd}|jj|}|s||_||j|<n'|jd|j|_||_|j j|}|r |j s|jtj|j||j |<y)z+Add *event* to the pending batch for *key*.rMrN) r rOr[r*_last_chunk_lenr\r=r>r? create_task_flush)rr`r chunk_lenexistingpriors renqueuezTextBatchAggregator.enqueues (b) ==$$S)$-E !!&DMM# '}}oR |r1r[)rr;s r cancel_allzTextBatchAggregator.cancel_allsV''..0 D99;   !!# rr)r3r4r5r6r8r7rr:r_rr9rhrdrqrrrrRrRtsw.! # : :  :  :%D%I\III"///(rrRr)rrrrr)NrNN)Cr6r?rArploggingrrcrpathlibrtypingrrrgateway.platforms.eventrutilsr getLoggerr3rr to_threadrTaskrEr9r7rLrDOTALL MULTILINErQrVrXrrr:rrtrrrrrrrrrrrrr rrrrr)r0r-r.rIrKrNrOrPrRrrrr|s3   000#   8 $))^    HW\\2 t %~c3h/%c%#%C%TX%RZZ ")),e4RZZ bii(%0RZZ3RYY?GRZZ1299=uERZZ()2.RZZ U#RZZ r||,b1RZZ)*E2RZZ F#  U"U"p+++ RZZ STst !3!49! (T#Y(3(.332TX9=^b@D2B2BcDj2B,/$J2BI[2B&=2BI[2BtM#M$M 73747 +++ ?s?t? --- &2::&9:4"( s { Fcss  C$cK]2N$*/N(0$f0#*"0MdM, RRr