߲j&)dZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZej@dZ!de"fdZ#d Z$ejJd ejLe$d Z'e(hd Z)e)d hzZ*ejJdZ+ejJdZ,dZ-dZ.dZ/dZ0dddZ1dZ2dZ3de4de ejjfdZ6de4ddfdZ7de8de9fdZ:de4de e4fd Z;de4de4fd!Zd&e4d'eddfd(Z?d)e4d*e4d+e e4d,e4d-e4de4f d.Z@ddd/d+e e4d0eAd1e9d2e e e4d3e e e4d4e e e4d5e e8d6e e8ddfd7ZBGd8d9ZCddlDZDddlEZEy):zFull-text / trigram / CJK message search and FTS maintenance for SessionDB. Plain mixin for ``hermes_state.SessionDB`` (no ``__init__``/state of its own). Must never import hermes_state (cycle); shared constants live in hermes_state_common. N)AnyCallable CollectionDictListOptionalTuple)describe_skill_invocation)FTS_CJK_STALE_KEYFTS_SQL FTS_STALE_KEYFTS_STORAGE_VERSIONFTS_TOOL_CONTENT_PREFIX_CHARSFTS_TRIGRAM_EXCLUDED_SOURCESFTS_TRIGRAM_SQLMAX_FTS5_QUERY_CHARSSCHEMA_VERSION_FTS_CJK_TRIGGERS escape_likefts_rebuild_admissionfts_trigram_session_sqlrouted_sessions_setting hermes_statereturnctdd} |t|jdk(rdSt|S#tt f$rYywxYw)zU``sessions.search_slow_ms`` for the served profile (default 1000; 0 logs every call).search_slow_msHERMES_SEARCH_SLOW_MS@@)rstrstripfloat TypeError ValueError)values 8/home/ubuntu/.hermes/hermes-agent/hermes_state_search.py_search_slow_msr'sS #$46M NE#e**:*:*<*BvTe T z "s9 9A  A z+{}():"^@/#&|~[]<>,;!?$=\'[]>ORANDNOTNEARz "[^"]+"|\S+z"[^"]*"zLm.timestamp, m.tool_name, s.source, s.model, s.started_at AS session_startedzCsubstr(m.content, max(1, instr(m.content, ?) - 40), 120) AS snippetz`(m.content LIKE ? ESCAPE '\' OR m.tool_name LIKE ? ESCAPE '\' OR m.tool_calls LIKE ? ESCAPE '\')z(COALESCE(m.content, '') LIKE ? ESCAPE '\' OR COALESCE(m.tool_name, '') LIKE ? ESCAPE '\' OR COALESCE(m.tool_calls, '') LIKE ? ESCAPE '\')zORDER BY m.timestamp DESC, rankzORDER BY m.timestamp ASC, rank)newestoldesta%WITH target AS ( SELECT session_id, timestamp, id FROM messages WHERE id IN ({ids}) ) SELECT t.id AS match_id, m.role, m.content FROM target t JOIN messages m ON m.id IN ( t.id, (SELECT p.id FROM messages p WHERE p.session_id = t.session_id AND (p.timestamp, p.id) < (t.timestamp, t.id) ORDER BY p.timestamp DESC, p.id DESC LIMIT 1), (SELECT n.id FROM messages n WHERE n.session_id = t.session_id AND (n.timestamp, n.id) > (t.timestamp, t.id) ORDER BY n.timestamp, n.id LIMIT 1) ) ORDER BY t.id, m.timestamp, m.id))iNi)i4iM)iiߦ)i0i?0)i@0i0)i0i0)iikeycD|jd|fjS)z9Point-read one ``state_meta`` row (``None`` when absent).z*SELECT value FROM state_meta WHERE key = ?)executefetchone)connr0s r& _meta_rowr5Ps <<Dsf M V V XXkeysc\|jddjd|Dd|y)Nz%DELETE FROM state_meta WHERE key IN (,c3 K|]}dyw?N.0_s r& z_delete_meta..VsAT!#AT ))r2join)r4r7s r& _delete_metarEUs,LL8ATtAT9T8UUVWY]^r6cpc4tfdtDS)Nc3DK|]\}}|cxkxr|kncywNr=)r?lohirFs r&rAz_is_cjk..Zs8&"brR~2~~8s )any _CJK_RANGES)rFs`r&_is_cjkrNYs 8K8 88r6 raw_querycv|jDcgc]}|jtvs|c}Scc}wrI)splitupper_FTS_OPERATORS)rOts r&_non_operator_tokensrU]s* ( L!AGGI^,KA LL Ls66cNdjd|jDS)zYQuote each non-operator token (neutralising FTS5 special characters), keeping AND/OR/NOT. c3|K|]4}|jtvr|nd|jddzdz6yw)"z""N)rRrSreplace)r?toks r&rAz$_quote_fts_tokens..cs=Y\syy{n,# C8N2NQT2TTs:<)rDrQ)rOs r&_quote_fts_tokensr\as) 88`i`o`o`q r6termc(dt|dgdzS)z;One ``%term%`` bind per column of ``_LIKE_ANY_COLUMN_SQL``.%) _escape_like)r]s r& _like_paramsrbhs T"#1 % & **r6decodedc2t|tro|Dcgc]:}t|ts|jddk(s)|jdd<}}dj d|Dj xsdSt|t r|SdScc}w)zVMultimodal part list -> joined text (or the placeholder); str passes through; else ''.typetextrrWc3&K|] }|s| ywrIr=r?rTs r&rAz _flatten_text..qs.aA.sz[multimodal content]) isinstancelistdictgetrDr!r )rcppartss r& _flatten_textroms'4 ,3gqz!T7JquuU[}`fOfvr"ggxx.5..446P:PP #.76B6hsBBBnamer%ct|tst|tst|d|dkrt |dy)Nz must be an integerrz must be greater than zero)riboolintr#r$)rpr%s r& _positive_intrtusG%j&<4& 3455 zD6!;<==r6 snippet_sqlfrom_sqlwhereorder_by limit_sqlc Rd|dtd|ddj|d|d|d S)zIResult-row SELECT shared by the FTS and LIKE routes (SQL text is pinned).zC SELECT m.id, m.session_id, m.role, z, z FROM zF JOIN sessions s ON s.id = m.session_id WHERE  AND z z )_SEARCH_SELECT_TAILrD)rurvrwrxrys r&_search_select_sqlr}|sX=!'()<<&'( J K   r6)after_ts before_tsparamsinclude_inactive source_filterexclude_sources role_filterr~rcv|s|jd|jd|<|jddjd|Dd|j||<|jddjd |Dd|j||r<|jd djd |Dd|j||+|jd |jt||,|jd |jt|yy)aAppend the visibility/source/role/session-start predicates every search route shares. Live rows (active=1) AND compaction-archived rows (compacted=1) are discoverable; only rewind/undo rows (active=0, compacted=0) are hidden. ``after_ts``/``before_ts`` bound ``sessions.started_at`` (inclusive / exclusive) inside the query so LIMIT cannot be filled by out-of-window hits.z!(m.active = 1 OR m.compacted = 1)z(COALESCE(m.display_kind, '') <> 'hidden'Nz s.source IN (r9c3 K|]}dywr;r=r>s r&rAz)_search_filter_clauses..s-Iac-IrBrCzs.source NOT IN (c3 K|]}dywr;r=r>s r&rAz)_search_filter_clauses..s1O!#1OrBz m.role IN (c3 K|]}dywr;r=r>s r&rAz)_search_filter_clauses..s+EAC+ErBzs.started_at >= ?zs.started_at < ?)appendrDextendrs)rwrrrrrr~rs r&_search_filter_clausesrs  89 LL;<  }SXX-I=-I%I$J!LM m$" (1O1O)O(PPQRS o& {388+E+E#E"FaHI k" () c(m$ '( c)n%r6ceZdZdZdZedeeedee edffdZ dydZ dee ee ffd Zdee ee ffd Zd edee ee ffd Zd ZdZdeddZdedZdedZdydZdydZd edee eefddfdZdefdZdefdZddd edeedededef dZdefd Z dyd!Z!dyd"Z"defd#Z#dyd$Z$e%defd%Z&e%dyd&Z'dd'd(ede(fd)Z)dyd*Z*defd+Z+de(fd,Z,d-eddfd.Z-defd/Z.deefd0Z/dd1d2d3ee0e ee fgdfd4ede ee ffd5Z1 dzd6ed7e(d8e(d9e(d:ee edfde ee ff d;Z2 d{d6edZ3e%d?edefd@Z4e%dAedefdBZ5e%dAede(fdCZ6e%d?edefdDZ7e%d?edeefdEZ8e%d?edefdFZ9edGedefdHZ:dGedefdIZ;d?edefdJZdddPdKedLedMedQeedReedeee ee ff dSZ?dTeedUe=dVedWedee ee ff dXZ@e%d?ede eee eeffdYZAd?edSee module docstring — mixin for SessionDB (Search cluster).) id session_idrolesnippet timestamp tool_namesourcemodelsession_startedcontextfieldsr.c|yt|tr tdt|j |j }|r&t ddjt|tfd|j DS)z=Validate and canonically order an optional result projection.Nz?search fields must be a collection of field names, not a stringz unknown search result field(s): , c3,K|] }|vs| ywrIr=)r?field requesteds r&rAz.s`uUV_M_U`s ) rir r#set difference_SEARCH_MESSAGE_RESULT_FIELDSr$rDsortedtuple)clsrunknownrs @r&_search_message_fieldsz)SessionSearchMixin._search_message_fieldss~ > fc "]^ ^K &&s'H'HI ? &QX/@Z?[\] ]`(I(I```r6Nc|jsy |j|jy#t$r }tj d|Yd}~yd}~wwxYw)uOne bounded FTS5 merge pass that never fails the already-committed write (a caller must never replay an ambiguous, possibly-durable write — even on the bare SystemError CPython's sqlite3 layer can raise under cross-thread errmsg scrambling).N) max_pagesz-FTS incremental merge failed after commit: %s) _fts_enabled_merge_fts_incrementally_FTS_MERGE_MAX_PAGES_PER_INDEX Exceptionloggerwarning)selfexcs r&_try_incremental_merge_ftsz-SessionSearchMixin._try_incremental_merge_ftssS   Q  ) )D4W4W ) X Q NNJC P P  Qs, AAAc$|jdS)zDeferred-rebuild progress ``{"pending", "total", "indexed", "percent"}`` or None. Reads via the pooled reader (not get_meta/self._lock) so search never blocks on the writer. fts_rebuild_rebuild_statusrs r&fts_rebuild_statusz%SessionSearchMixin.fts_rebuild_statuss##M22r6c$|jdS)z:CJK-index backfill progress, or None when none is pending.fts_cjk_rebuildrrs r&fts_cjk_rebuild_statusz)SessionSearchMixin.fts_cjk_rebuild_statuss##$566r6prefixc N|jd|d|df}|Dcic] }|d|d }}|j|d}|t|dkryt|t|j|dxsd}}d||tdtd|z|z d Scc}w) Nz5SELECT key, value FROM state_meta WHERE key IN (?, ?) _high_water _progressr0r%rTd)pendingtotalindexedpercent) _read_allrlrsmin)rrrowsrmeta high_waterrprogresss r&rz"SessionSearchMixin._rebuild_statuss~~U"(5&7KLN.23%!G*$33XX 45  ZA!5j/3txx6()8L/M/RQR+Sx%HQTUXZ]^adl^lot^tZuQvww 4sB"zINSERT INTO {table}(rowid, content, tool_name, tool_calls) SELECT m.id, m.content, m.tool_name, m.tool_calls FROM messages m WHERE m.id > ? AND m.id <= ? {extra}AND NOT EXISTS (SELECT 1 FROM {table}_docsize d WHERE d.id = m.id)a4INSERT INTO messages_fts(rowid, content, tool_name, tool_calls) SELECT m.id, CASE WHEN m.role = 'tool' THEN substr(COALESCE(m.content, ''), 1, ?) ELSE m.content END, m.tool_name, m.tool_calls FROM messages m WHERE m.id > ? AND m.id <= ? AND NOT EXISTS (SELECT 1 FROM messages_fts_docsize d WHERE d.id = m.id)zINSERT INTO messages_fts_trigram(rowid, content, tool_name) SELECT m.id, m.content, m.tool_name FROM messages m JOIN sessions s ON s.id = m.session_id WHERE m.id > ? AND m.id <= ? AND m.role <> 'tool' AND szP AND NOT EXISTS (SELECT 1 FROM messages_fts_trigram_docsize d WHERE d.id = m.id)zINSERT INTO {table}(rowid, content, tool_name, tool_calls) SELECT id, CASE WHEN role = 'tool' THEN substr(COALESCE(content, ''), 1, zW) ELSE content END, tool_name, tool_calls FROM messages WHERE id > ? AND id <= ?{extra}c|jdfg}|jr|j|jdf|j d|t j dy)aQFinalize the deferred rebuild: boundary sweep + clear markers. The sweep is cheap insurance against a write that slipped between high_water capture and trigger activation. The trigram half is gated on ``_trigram_available``: without the tokenizer/table an unconditional INSERT raises and aborts the whole rebuild.TFru7Deferred FTS rebuild complete — all messages indexed.N)_BASE_BOUNDARY_SWEEP_SQL_trigram_availabler_TRIGRAM_BOUNDARY_SWEEP_SQL_rebuild_finishrinfo)rsweepss r&_fts_rebuild_finishz&SessionSearchMixin._fts_rebuild_finishsT 00$78  " " MM4;;UC D ]F3 MNr6c|jjdd}|jd|dfgd|_tj dy) z?Boundary sweep + clear the cjk markers; index becomes servable.messages_fts_cjkzAND m.role <> 'tool' tableextrarFTu7CJK FTS index backfill complete — serving CJK search.N)_BOUNDARY_SWEEP_SQLformatr_fts_cjk_availablerr)rsweeps r&_fts_cjk_rebuild_finishz*SessionSearchMixin._fts_cjk_rebuild_finishsL((//6HPg/h .%0@A"& MNr6 sweep_sqlsc6fd}|j|y)zSweep a generous window around the high-water boundary, then clear the markers. ``(sql, bounded)``: a bounded sweep takes the tool-content prefix_chars param first.ct|d}|Bt|d}D]/\}}|rtfnd}|j|g||dz |dz1t |ddy)Nrrr=ir)r5rsrr2rE)r4hw_rowhwsqlboundedrrrs r&_doz/SessionSearchMixin._rebuild_finish.._dostx{%; 'tool'rrz-CJK FTS rebuild chunk failed (will retry): %s)rr)r_fts_cjk_loadedrrrr)rinserts r&fts_cjk_rebuild_stepz'SessionSearchMixin.fts_cjk_rebuild_step0s_  (<(<''..5GOd.e!!"3fXdFbFb+Z"\ \r6F)r insert_sqlsrrc~ |jd}|yt| |j fd} |j|}|dur-|j} |r dks | | d| dk\r|yt|S#tj $r } t j|| Yd} ~ yd} ~ wwxYw) zShared chunk engine for the base and CJK deferred backfills. ``finish_when_empty`` finalizes a high_water <= 0 marker (empty messages table) instead of leaving it pending.rNFct|d}|yt|d}|k\ryt|z}D]}|j|||f|jdt |df|kS)NrFrz-UPDATE state_meta SET value = ? WHERE key = ?)r5rsrr2r ) r4rowrrRrchunkrrrs r&rz-SessionSearchMixin._rebuild_step.._doBsDVHI"67C{3q6{H:%5(*5E" 5 S8U"34 5 LLH3u:Z`YaajWkJl m:% %r6Trrr) get_metars_FTS_REBUILD_CHUNK_ROWSrsqlite3OperationalErrorrdebugrrr) rrrrrrhigh_water_rawrmorerstatusrrs `` @@r&rz SessionSearchMixin._rebuild_step8s&'=>  !( ,, &" &&s+D 5=))&1F!jAo"vi'8F7O'KDz''  LL3 ' sB B<B77B<cj5}|jdjjdddzfj Dcgc]}|d }}dddsy|dfd}dd t t d tffd  tj|Scc}w#1swYRxYw#tj$r }tjd |Yd}~y d}~wwxYw)uTear down one chunk of a demoted v22 FTS shadow table (a PLAIN table now); True while work remains. INTEGER single-column-key tables drain with a high-water marker so each chunk's scan is bounded (restarting the scan was O(n²)); compound-key tables keep the chunked ``LIMIT`` delete — they are small by construction. Single-column-key trash tables (the common shape — FTS shadow tables carry a rowid/integer PK) are drained with a high-water marker mirroring :meth:`fts_rebuild_step`: each chunk deletes only rows after the previously-drained key, so the per-chunk scan is bounded instead of re-scanning from the start of the table every chunk (O(n²) total on large trash tables, #79324). zNSELECT name FROM sqlite_master WHERE type = 'table' AND name LIKE ? ESCAPE '\'r@z\_r_rNFc |jd dDcgc]'}|ddkDs |d|dxsdjf)}}|rdjd |Dnd }t|dk(r|ddd k(rd d }t ||}|t |dnd}|jd|d d|d|d j |fj}|s  ||S|dd}|jd d|d|d||f} | jdkDr j|t||y|jd d|d|d d j d } | jdk(r |SdScc}w)NzPRAGMA table_info(rCrrrc3&K|] \}}| ywrIr=)r?rp_typs r&rAzKSessionSearchMixin._fts_teardown_trash_step.._do..ws;ZT4D;srowidINTEGER fts_teardown_rzSELECT z FROM z WHERE z > ? ORDER BY z LIMIT z DELETE FROM z > ? AND z <= ?cursorTz WHERE (z ) IN (SELECT ) r2rRrDlenr5rsrfetchallrowcountset_metar ) r4rpk_infor0 marker_keyrr upper_rowsrRcur_droprtbls r&rz8SessionSearchMixin._fts_teardown_trash_step.._dous=A\\L^_b^ccdJe=fsjklmjnqrjr!qtzr0023sGs?F$));7;;GC7|q WQZ]i%? -SE; j1,/OSQ[ !\\cU&WSEuGTXTpTpSqrM(*" z22"2q)ll\#gcU)C5PU#VYcejXkl<._drops9 LL06 7%T:. KK< HSM T  ++C01 1_  b''  LLKS Q s6AB4 B/!B4C/B44B=C3C..C3ch|jsyd}|j|r|jyy)zFrom-scratch rebuild of a stale cjk index (triggers were dropped, gap extent unknown): drop table + triggers, clear the breadcrumb, recreate (fresh backfill markers).Nct|tytD]}|jd||jd|jdt |tddy)NFzDROP TRIGGER IF EXISTS z%DROP TABLE IF EXISTS messages_fts_cjkz(DROP VIEW IF EXISTS messages_fts_cjk_srcfts_cjk_rebuild_high_waterfts_cjk_rebuild_progressT)r5r rr2rE)r4trigs r&rz7SessionSearchMixin._fts_cjk_reset_if_stale.._dosd019) ? 6tf=> ? LL@ A LLC D 02NPj kr6)rr_ensure_cjk_schema_committedrrs r&_fts_cjk_reset_if_stalez*SessionSearchMixin._fts_cjk_reset_if_stales6##     s #  - - / $r6c|j5|j|j|jjdddy#1swYyxYwrI)_lock_ensure_fts_cjk_schema_conncommitrs r&r#z/SessionSearchMixin._ensure_cjk_schema_committeds@ ZZ  ' ' 3 JJ      s 6A  Ac |jdjdsy|jdjd S#tj$rYywxYw)aTrue when the base FTS table indexes nothing while ``messages`` has rows (post-demote empty-index shape). Caller holds ``self._lock``. docsize is the authoritative "is this rowid indexed" surface; EXISTS not COUNT(*) because this runs on every writable open.z%SELECT EXISTS(SELECT 1 FROM messages)rFz1SELECT EXISTS(SELECT 1 FROM messages_fts_docsize))r2r3rrrr4s r&'_fts_external_index_empty_with_messagesz:SessionSearchMixin._fts_external_index_empty_with_messagess` << GHQQSTUV||$WXaacdeff f''  s"A"AAAct|d7|j|s|j||jdd|yy)ahigh_water without progress: fts_rebuild_step reads missing progress as "done by another process" and optimize would no-op then stamp. Reset to known-empty, re-seed. Truncation goes through FTS5 ``'delete-all'`` (a plain DELETE is O(rows) and corrupts the index when indexed rows diverged from ``messages``); the backfill worker replays without an anti-join, so it needs a known-empty index. A missing docsize table counts as empty.fts_rebuild_progressN0r )r5_fts_index_known_empty_reset_fts_index_to_emptyrr,s r&_reseed_missing_progressz+SessionSearchMixin._reseed_missing_progresssF T1 2 :..t4..t4 MM0#dM C ;r6c t|jdjddk(S#tj$rYywxYw)zZTrue when the base external-content index holds no rows (a missing table counts as empty).z)SELECT COUNT(*) FROM messages_fts_docsizerT)rsr2r3rr)r4s r&r1z)SessionSearchMixin._fts_index_known_emptysI t||$OPYY[\]^_cdd d''  s-0AAc dD]F}tjtj5|j d|d|ddddHy#1swYSxYw)zTruncate the v23 external-content tables via FTS5 ``'delete-all'`` (O(1); a plain DELETE is O(rows) and corrupts the index when indexed rows diverged from ``messages``).)rmessages_fts_trigram INSERT INTO (z) VALUES('delete-all')N) contextlibsuppressrrr2)r4rs r&r2z,SessionSearchMixin._reset_fts_index_to_emptysd< OC$$W%=%=> O |C5#6LMN O O O O Os A  A forcer<ct|d}|!|s|j|t|dS|jdj d}|j dt |||j dd|t|S)zWrite ``fts_rebuild_high_water`` / ``fts_rebuild_progress`` for a full backfill; returns the high-water id. Without ``force`` an existing high_water only gets a missing progress key repaired. Caller holds the write transaction.fts_rebuild_high_waterrz)SELECT COALESCE(MAX(id), 0) FROM messagesr r/r0)r5r3rsr2r3rr )rr4r< existing_hwrs r&_seed_fts_rebuild_markersz,SessionSearchMixin._seed_fts_rebuild_markerss &>?  "5  ) )$ /{1~& & \\E F O O QRS T .B E ,c$ ?2wr6c2fd}j|y)uHeal interrupted demote/backfill bookkeeping before optimize runs: orphan high_water gets progress re-seeded; an empty external index with messages and no markers gets a full backfill claim. Never invents markers on a still-legacy inline DB — optimize would then skip demote and INSERT against the inline table forever. Covers two post-#65798 failure classes: ct|dj|yj|ryj|r t |dj |dyy)Nr>fts_storage_versionTr;)r5r3_db_has_legacy_inline_ftsr-rEr@)r4rs r&rz._dose78D--d3--d3;;DAT#89..t4.@Br6Nrr$s` r&_repair_optimize_bookkeepingz/SessionSearchMixin._repair_optimize_bookkeepings A C r6ct|jr |jry|j5}|j|xsft |dduxsV|j xr"t |dduxst |t duxs$|j|xs|j| cdddS#1swYyxYw)aNTrue when `optimize_fts_storage()` has work: legacy inline FTS or a v23 trigram still carrying ``tool_calls`` (``_db_needs_fts_storage_upgrade``), an interrupted optimize (markers/trash), a CJK backfill on this tokenizer-capable host, or an empty external index without markers. False when FTS5 is unavailable.Fr>Nr ) r read_onlyr_db_needs_fts_storage_upgrader5rr _has_fts_trashr-r,s r&fts_optimize_availablez)SessionSearchMixin.fts_optimize_availables   DNN ^^  2248FT#;SessionSearchMixin._demote_legacy_fts_to_trash.._stages  # #D ) LLG H||?hj 89 p ;<<<L (* [CLL<Ax7PQTUVQWPX!YZ [//D/AB 7 8Ir6z@failed to create v23 messages_fts during optimize-storage demote)rsr_ensure_v23_fts_tables)rrNrs` r&_demote_legacy_fts_to_trashz.SessionSearchMixin._demote_legacy_fts_to_trashs3  >$$V, - ##$fg r6failure_messagecL|j5|j|jdt}|j|jdt}t ||_|stj||jjdddy#1swYyxYw)zEnsure the v23 base + trigram tables under the lock (IF NOT EXISTS, cheap); raise *failure_message* without the base table (the backfill loop would retry forever).rr6N) r'_ensure_fts_schemar)r rrrrrrr*)rrQbase_ok trigram_oks r&rOz)SessionSearchMixin._ensure_v23_fts_tables>sZZ --djj.'RG00=SUdeJ&*:&6D #..?? JJ       s BBB#c |j5|jjddddd} |j5|jjdddd|S#1swY>xYw#tj$r"}t j d|d}Yd}~od}~wwxYw#1swY|SxYw#t$r!}t jd|Yd}~|Sd}~wwxYw)zPhase 3: reclaim freed pages to the OS. False when VACUUM failed (usually no free disk for its temp copy; a later VACUUM reclaims).VACUUMNTz$VACUUM after FTS optimize failed: %sFzPRAGMA wal_checkpoint(PASSIVE)z9WAL checkpoint (PASSIVE) after optimize VACUUM failed: %s) r'r)r2rrrrrr)r vacuum_okrs r&_optimize_vacuumz#SessionSearchMixin._optimize_vacuumIs  - ""8, -I [ E  ""#CD  E% - -''  NNA3 GI  E [ LLTVY Z Z [sb A3A' A3 B8B+B8'A0,A33B(B##B(+B50B85B88 C"CC"ct|dy|j|ry|j|ry|jdt t |t |d|jdttfy) aPhase 4 (inside the write transaction, so a concurrent writer cannot race a stamp past incomplete work): stamp the FTS layout (source of truth for "optimized"), clear the "available" flag, advance a lagging schema_version. Returns a refusal reason or None. Refuses while optimize work remains; an empty base index against non-empty messages also refuses (settling there meant permanent search-index loss).r>Nbackfill_incompleteteardown_incompleterCr rJz7UPDATE schema_version SET version = ? WHERE version < ?) r5rIr-rr rrEr2rr,s r&_optimize_settlez#SessionSearchMixin._optimize_settleast T3 4 @(   t $(  7 7 =( +S1D-Ed ST34 NQ_aoPpqr6T) progress_cbvacuumr^r_c jsdddSjrdddSjj5j j }dddj ddu}r|sjn|r|sjdjjrjdtd dffd dtd df fd } d |d j d |d j d |d jj!5}t#|ddu}j%|}j'|} dddss r'|s rdnd} t(j+d| | d| ddSd} |r dj-} j/j0} | t(j+d| d| | dS dt(j3dt4d| dS#1swYxYw#1swYxYw)aqRepair an older FTS layout into the current v23 shape, foreground and to completion: legacy-v22 inline -> external-content, or a v23 ``messages_fts_trigram`` that still stores ``tool_calls``. Re-running resumes. ``progress_cb`` receives {"phase", "percent", "indexed", "total"}. A missing trigram tokenizer is not fatal (CJK falls back to LIKE).Ffts5_unavailable)okreasonrGNr>z?failed to re-create v23 messages_fts on optimize-storage resumephasercyjxsj}||r|dnd|r|dnd|r|dnddy)Nrrrrr)rdrrr)rr)rdstr^rs r&_emitz6SessionSearchMixin.optimize_fts_storage.._emitsZ"((*Kd.I.I.KB %RByMS57ByMQXZG`ac dr6c tj}|sy|tjtjtj|z j zp)uRun *step* to completion; the inter-chunk sleep is the single place the duty cycle is enforced — back-to-back BEGIN IMMEDIATE chunks starve a live gateway/CLI out of its lock retries.N)time monotonicsleepmax_FTS_REBUILD_MIN_PAUSE_FTS_REBUILD_DUTY_FACTOR)rdstep_t0rgrs r&_drivez7SessionSearchMixin.optimize_fts_storage.._drivesZnn&ve  3t::T^^=MPS=SW[WtWt > >>;7 7 ))+ ZZ S$($F$Ftzz$R ! S-- 89E   , , . 2  ' '(i j $$&     - - / d d d w# w w jz4001 jz4445 jz4889 ^^  M%d,DETQM--d3KFFtLK M K;.;{*PeF NNm!=+{ L6tD D  (O--/I%%d&;&;<   NNI7 S7 J J f  EGZ[ 22 S ST M MsH01H=0H:=Iraround_message_idwindowbookend keep_rolesc  tdj||}|d}|sgddggdS|}|Bt|} |D cgc],} | jd|k(s| jd| vs+| .}} g} g} dkDr|dnd d j d |Dd |gn t |j 5d tdtdtffd } | d|ddd} t t| d|ddd} ddddtttfffd }||d|d| Dcgc] }|| c}| Dcgc] }|| c}dScc} w#1swY]xYwcc}wcc}w)aAnchored window (``get_messages_around``) plus session bookends, so one call yields the goal and the resolution of a long session. ``window`` is filtered to ``keep_roles`` (None disables) EXCEPT the anchor; ``bookend_start`` / ``bookend_end`` are the first/last ``bookend`` non-empty-content messages with ids strictly outside the window (empty when it overlaps the head/tail). Empty result when the anchor isn't in the session.r)r}r})r}messages_beforemessages_after bookend_start bookend_endNrrrz AND role IN (r9c3 K|]}dywr;r=r>s r&rAz7SessionSearchMixin.get_anchored_view..sQjZ[RUQjrBrCop boundary_idorderc fjd|dd|d|gjS)Nz3SELECT * FROM messages WHERE session_id = ? AND id z ?z% AND length(content) > 0 ORDER BY id z LIMIT ?)r2r)rrrr~r4 role_clause role_paramsrs r&_bookendz6SessionSearchMixin.get_anchored_view.._bookendsS<<779d"[MJ'',gX7$[H;HH  hj !r6<ASC>r DESCrc,j|ddS)Nget_anchored_viewF) warn_context summary_flag)_row_to_message_dict)rrs r&_hydratez6SessionSearchMixin.get_anchored_view.._hydrates,,S?Raf,g gr6rr) rlget_messages_aroundrrlrDrjrr rsreversedrr)rrr|r}r~r primitive window_rowsfiltered_windowkeep_setmbookend_start_rowsbookend_end_rowsrrrr4rrs`` ` @@@r&rz$SessionSearchMixin.get_anchored_viewsgq/,,Z9JSY,Z )  Q!^`qst t%  !:H*5wQtHY9Y]^]b]bci]jnv]vqwOw(*&( Q; * 2".Qj_iQjIjHkkl8mK * 2"Z8HK! `T!!3!s!!&.c;q>$3G%O"#'#{2t?TV\1](^#_  ` hT#s(^ h&)DU:V'(893EFahqkF1ABAHQKB   +x ` `$GBs% ,E*6E*AE/E; F /E8limitrcb|rdnd}d}|j5}|jd||d|t|dzdzfj}dddd d lm}g} D]} t | t|k\r| S|j| d } |j| rCt| trt| xs| } n t| } d j| j} t | d kDr| dddz} | j| d| d| d| S#1swYxYw)uThe *limit* most-recent real user turns, newest first, as ``{id, timestamp, preview}`` (80 chars, whitespace collapsed); used by /rewind and ``/undo [N]``. Bookkeeping rows (``display_kind`` set) are excluded. Legacy compaction handoffs are role='user' rows with NO display_kind — invisible to SQL — so fetch with headroom and drop them in the decode loop; otherwise ``/undo N`` pairs an in-memory count that excludes handoffs with a DB pick that includes them.rz AND active = 1zJ AND (display_kind IS NULL OR display_kind = '' OR display_kind = 'steer')zRSELECT id, timestamp, content FROM messages WHERE session_id = ? AND role = 'user'z ORDER BY id DESC LIMIT ?rrNr)ContextCompressorcontentrWPMz...rr)rrpreview)rr2rsragent.context_compressorrr_decode_content_is_context_summary_contentrir r rorDrQr) rrrr active_clausedisplay_clauser4rrresultrrcrs r&list_recent_user_messagesz,SessionSearchMixin.list_recent_user_messagessR/4E f ^^  <<d /.!12++SZ!^a/0  hj   ?') `C6{c%j( **3y>:G <._hold$s3   ,3}-12$7 7r6rYrWr_z\*+*z(^|\s)\*z\1z(?i)^(AND|OR|NOT)\b\s*rz(?i)\s+(AND|OR|NOT)\s*$z\b(\w+(?:[._-]\w+)+)\bz"\1"rr) rr _QUOTED_PHRASE_REsubrZ_FTS5_SPECIAL_REr _contains_cjkrer! enumerate)rr sanitizediquotedrs @r&_sanitize_fts5_queryz'SessionSearchMixin._sanitize_fts5_querys8 ++,!  8_ 8 8&))%7??SI %((i8 ) $6$D$DY$O!))#s3IFF63 2 FF;y9 FF4b)//:KL FF5r9??;LM FF4gyI "=1 CIAv!))E!D/6BI C  r6rfc&td|DS)Nc3DK|]}tt|ywrIrNordr?chs r&rAz3SessionSearchMixin._contains_cjk..?s373r7#3s )rLrfs r&rz SessionSearchMixin._contains_cjk=s3d333r6c&td|DS)Nc3JK|]}tt|sdyw)rNrrs r&rAz0SessionSearchMixin._count_cjk..Cs8ws2w'718s##)sumrs r& _count_cjkzSessionSearchMixin._count_cjkAs8t888r6cdd}|D]%}tt|r|dz }|dk(ryd}'|dk(S)uTrue when any maximal CJK run is a single char: the cjk-bigram index stores unigrams only for isolated chars, so such a term can't match inside longer runs — keep LIKE.rrTr)rrunrs r&_has_lone_cjk_runz$SessionSearchMixin._has_lone_cjk_runEsG Bs2wq!8  axr6cg}tj|D]/}|j}|dvry|dk7s|j|1t |dk\rdj |SdS)agThe sanitized implicit-AND query rewritten as an any-term OR query, or ``None`` when relaxation does not apply: fewer than two searchable units (a single term cannot relax) or explicit ``OR``/``NOT`` (the caller expressed exact semantics). Quoted phrases stay whole units: ``"docker networking" tls`` -> ``"docker networking" OR tls``.>r*r,Nr+r OR )_LIKE_TOKEN_REfindallrRrrrD)runits raw_tokenrRs r&_or_relaxed_queryz$SessionSearchMixin._or_relaxed_querySsm '//6 (IOO%E %~ Y'  ( &)Z1_v{{5!>$>r6ct|jdj}t|xrtd|DS)zTrue when every non-operator token is >=3 chars: a shorter token produces no trigrams, and with FTS5's implicit AND one such token empties the whole MATCH.rYc38K|]}t|dk\ywr`N)rrhs r&rAz>SessionSearchMixin._trigram_eligible_tokens..gs#@ACFaK#@s)rUr!rrall)rtokenss r&_trigram_eligible_tokensz+SessionSearchMixin._trigram_eligible_tokensbs:&ekk#&6&<&<&>?F|@#@#@ @@r6rOc>tfdt|DS)uTrue when any non-operator CJK token has fewer than 3 CJK chars — the trigram tokenizer needs >=3 per token, so such a query must take the LIKE route.c3jK|]*}j|sj|dk,ywr)rr)r?rTrs r&rAz:SessionSearchMixin._has_short_cjk_token..ms.jQUXUfUfghUi3>>!$q(js33)rLrU)rrOs` r&_has_short_cjk_tokenz'SessionSearchMixin._has_short_cjk_tokenisj2Fy2Qjjjr6cn|j|dk\xr |j| xr |jS)uPer-token CJK length gate for the trigram index: ``广西 OR 桂林 OR 漓江`` has 6 CJK chars total but 2 per token, so trigram returns 0.r`)rrr)rrOs r&_trigram_route_okz$SessionSearchMixin._trigram_route_okos> *a/,8Q8QR[8\4\,++ -r6c4 |jry|j|xsd}|sy|j|sy|jdj}|jr|j |sy|j |rdSdS#t$rYy wxYw) z>Best-effort name of the routing path a query takes (log-only).like_scan_fts_staleremptyfts5rYfts_cjktrigram like_scanr) _fts_stalerrr!rrrr)rrrraws r&_describe_search_pathz(SessionSearchMixin._describe_search_pathus ,11%+2>I%%i0//#&,,.C&&t/E/Ec/J $ 6 6s ;9 L L  s, B B B >>', '<<<', '...', 40) AS snippetz' JOIN messages m ON m.id = z.rowidLIMIT ? OFFSET ?)rrr}) rrrrrfiltersrwrrs r&_fts_match_sqlz!SessionSearchMixin._fts_match_sqlsu7(#$#}uf88 ufo& ugF Gg=eWF KUT`bt F{r6 fail_openoperational_debugrrc |j|||fi|\}} |j||D cgc] } t| c} Scc} w#tj$r|rt j |dYytj$r5} ||j| st jd|| Yd} ~ yd} ~ wwxYw)uRun one MATCH against *table*; ``None`` when the query cannot execute (tokenizer / syntax) so the caller falls back. *fail_open* names the index for the substring-capable routes: a corruption-class ``DatabaseError`` there detaches the derived indexes (``_enter_fts_fail_open``) and answers from canonical rows — a live search never runs the unbounded rebuild. Other ``DatabaseError``s propagate.T)exc_infoNz[%s FTS search hit a corruption error (%s); detached FTS and falling back to canonical LIKE.) rrrkrrrr DatabaseError_enter_fts_fail_openr) rrrrrrkwargsrrrrs r& _match_rowszSessionSearchMixin._match_rowss*d))%lUfU V )-V)DE#DIE EE''   . >$$  (A(A#(F NNm3   s.AAAA,B=6B=+B88B=rwrrxrycttd|||}|j||Dcgc] }t|c}Scc}w)zDCanonical-table LIKE scan; ``params[0]`` is the snippet anchor term.z messages m)r}_LIKE_SNIPPET_SQLrrk)rrwrrxryrrs r& _like_rowszSessionSearchMixin._like_rowss9 !2L%S\]%)^^C%@AcS AAAs=cgg}d}tj|D]}|j}|dk(r|dr|jgd}1|dvr6|dk(rd}>|j dj dj }|so|dj||fd}g}g}d }|D]} | rt d | Dsg} | D]G\}} | j| r d t nt |jt||C| rF|}I|jd d j| ddj|||fS)zCompile the supported FTS boolean subset into LIKE predicates: terms within an OR group are ANDed (FTS5's implicit conjunction) and ``NOT`` negates the next term.Fr*r >r+r-r,TrYrNc3(K|] \}}|  ywrIr=)r?r@negateds r&rAzASessionSearchMixin._compile_like_boolean_query..s#HJAwK#HszNOT r8r{rCr) rrrRrr!rL_LIKE_COALESCED_COLUMN_SQLrrbrD) rgroups negate_nextroperatorr]compiled_groupsr snippet_termrclausesrs r&_compile_like_boolean_queryz.SessionSearchMixin._compile_like_boolean_querysu13t '//6 $I (H4":MM"%# ?*5 " ??3'--c288:Dr !!4"56#  $"&(&*  AE#H%#H H!#G!& ( gg&@%ABSmn l401'#'L  (  " "Qw||G'<&=Q#? @ A{{?+V\AAr6sortc |j|\}}}|r|gSd|dg} t| |fi|t|tr#|j j dk(rdnd} |j | |g|||d| d| d S) z;Search canonical messages while derived FTS state is stale.r8rCr/rrzORDER BY m.timestamp z, m.id rrxry)rrrir r!lowerr) rrrrr r predicaterrrwrs r&_search_messages_like_fallbackz1SessionSearchMixin._search_messages_like_fallbacks+/*J*J5*Q' 6<L0IYKq!"uf88#D#.4::<3E3E3G83SY_u|&Lf&Le&LV&L*?wgeW(Uasu ur6c|js |jsy |jdtf}|d|_dx|_x|_|_yy#tj $rYywxYw)z@Observe fail-open initiated by another process sharing state.db.Nz.SELECT 1 FROM state_meta WHERE key = ? LIMIT 1TF)rr _read_oner rErrorrr)rstales r&_refresh_fts_stale_statez+SessionSearchMixin._refresh_fts_stale_statesr ??$"3"3  NN#SVcUefE  "DOTY YD  Y 7$:Q }}   sAA('A(matches result_fieldsc R|d|vrtdt|dD]}|||dz}|Dcic]}|dg }} tjdj d|D}|j 5}|j |t|j} ddd D]} || d j|  |D]M} |j|dgD cgc](} | d t|j| d dd d *c} |d<O|D]}|jd d|(|D cgc]}|D cic] } | |vs| || c} }}} |Scc}w#1swYxYw#t$ri}YwxYwcc} w#t$rg|d<YwxYwcc} wcc} }w)zOAttach neighboring messages in bounded batches, only when context is requested.Nrrirr9c3 K|]}dywr;r=r>s r&rAz>SessionSearchMixin._finalize_search_matches..sAX!#AXrB)idsmatch_idrr)rr)ranger_CONTEXT_WINDOW_SQLrrDrr2rjrrrrlrorpop) rrrstartbatchmatchcontextsrr4rrrs r&_finalize_search_matchesz+SessionSearchMixin._finalize_search_matchess  I$>q#g,4 .eck29>?E$KO??"-44AXxAX9X4YC)LT#||Ch@IIKL#> Z188=>#.E.(0||E$K'D,F #&)[]4K_K_`cdm`nKo=pquru=vw,Fi(. .& 'E IIi & '  $gno^cY%SX.uU|+YoGo+@LL!"!H",F%.+-i(. Zosp E# F# F F F#(E1 -E44 FFF  FFF#rrrr~rc tj} d} |j||||||||| | |  } | tj| z dz}|tk\r9tj d|j ||| t | nd|ddSS#tj| z dz}|tk\r9tj d|j ||| t | nd|ddwwxYw)z:meth:`_search_messages_impl` plus one log line per slow search with the routing path taken. Threshold HERMES_SEARCH_SLOW_MS (default 1000; 0 logs every call).N) rrrrrr rrr~rrzr|j@d|fi|xs|}|s5|jBr)|jE|r|j@d|fi|xs|}|s<|s:|js.|jG|}||j@d |fddd|xs|}|j || Scc}w#t(j*$rgcYSt(j,$r4}|j/|s|j|f|||d| }Yd}~d}~wwxYwcc}w#t(j*$r!}t8j;d |Yd}~d}~wwxYw)abFTS5 search across session messages (keywords, ``"phrases"``, AND/OR/NOT, ``prefix*``). Returns snippet + session metadata + 1-message context per hit; ``fields`` selects a projection. ``sort``: None = BM25 rank; "newest"/"oldest" = timestamp then rank (the CJK LIKE fallback ignores it). Rewound rows (``active=0, compacted=0``) are excluded by default; compaction-archived rows ARE included; ``include_inactive`` = every row. ``after_ts``/``before_ts`` bound ``sessions.started_at`` on every route (FTS5, CJK, trigram, LIKE fallback, unindexed-gap supplement).rrrrr~rtool)rrr )rNz ORDER BY rank)rrrc3,K|] }|tvywrI)r)r?srcs r&rAz;SessionSearchMixin._search_messages_impl..Ls'ePS/K(K'esrrc32K|]}|dvs |yw)rNr=)r?rseen_idss r&rAz;SessionSearchMixin._search_messages_impl..hsQQ49PqQs z$Unindexed-gap supplement skipped: %srYrr6z OR-relaxedz1OR-relaxed FTS retry failed; keeping empty resultrr=)$rr!rrkrr"rrr _FTS_ORDER_BYrlrir r rrrLr _search_cjkrrrrrrrr_search_unindexed_gaprrrr\rrrrr)rrrrrrrr rrr~rrrrrroutewants_unindexed_rowsis_cjkrrrr gap_matchesrfb_queryrelaxedr/s @r&r%z(SessionSearchMixin._search_messages_impl%s33F; EKKMI))%0I(8 '6K (I? 6[09d99%ruU[bfrjqrG00 0V V %%' ??9d99%ruU[bfrjqrG00 0V V  I$((DRUAV););)=\`bqr V,eFVgV!%[ 1 Kf 6K g   eC'eWd'e$e ##E* &&u.BEJG-$--neMuMKC w04sF0KL49LL"  " " $ 0S\E5I J8d88G @T`X_` -45AdG5Q+QQvtK/@V{EZ(S)9)?)?)ABH&&*$**+=xQ5Q\U\t664;X;XY^;_*$**+A8UuU`Y`vdoo,,U3G"*$**>74l=p4-24?7>,,WM,RRuM++  (( w005=$==ev5Y_fjvnuv w"6++ J CSII JsZL)L;L$M: M5 M:LM2,M2>)M--M25M::N. N))N.r4r3ct|jdj}t|}|jr.|s,|j|s|jd|fddd|}||S|j |r|s|jd|fddi|}||St |xs|g}|D cgc]}t|D]} | } }} d d jtgt|zd g} d D cic]} | ||  } } t| | fi| |j| |d g| |d|dddScc} }wcc} w)uCJK routing: the unicode61 table splits CJK into single characters (false positives, missed phrases). cjk-bigram serves every shape except queries wanting rows the substring indexes exclude (role='tool', cron/subagent sources) and LONE 1-char CJK runs (bigrams only exist for runs >=2 — LIKE is broader); then trigram (>=3 CJK chars per token); then a LIKE substring scan with one clause per non-operator token so "广西 OR 桂林 OR 漓江" matches each term.rYrz CJK-bigramz;messages_fts_cjk query failed; falling back to trigram/LIKErr6rTrigramr8rrCr*rrrORDER BY m.timestamp DESCrr ) r!r\rrrrrUrbrD_LIKE_ANY_COLUMN_SQLrrr)rrr4r3rOrr non_op_tokensr[rm like_params like_wherekrs r&r1zSessionSearchMixin._search_cjksKK$**, ' 2  " "+?H^H^_hHi&d&&"Kj;G"_jchjG"  ! !) ,5I&d&&'={iV_ichiG",Y7FI; *7S3cARSAQSQS S&++';&.s, lQ!3!3C!8!>!>!@ ls8:zm.id > ? AND m.id <= ?rrc3@K|]}t|D]}|ywrI)rb)r?r]rms r&rAz;SessionSearchMixin._search_unindexed_gap..s$=nD[ghl[m=nVWa=na=nsr;zLIMIT ?r ) rrrrR_LIKE_SKIP_TOKENSr<rrr) rrArrrr[termsrwrs r&r2z(SessionSearchMixin._search_unindexed_gaps((* >UaZI l.J`J`ajJk lBCIIK/@@BBI)R.B-Cc%j-PRy)6'?o=nE=nouf88uuQx&@&&@%&@Kf)24 4Bs!B2include_archivedrsourcesc T |xsdjj r|dkrgS|j|||t|dz|d|d }dtt t fdtf fd tt| fd  }|d |D cgc]\} } |  c} } Scc} } w) zSearch surfaced sessions by exact/prefix/substring session id. Also matches ``_lineage_root_id`` so an old compression root id resolves to the live continuation.rrT)rrHrrrrGorder_by_last_activeid_queryrrct|jdxsdt|jdxsdfDcgc]}|s|j}}tfd|Drytfd|DrdSdScc}w) Nrr_lineage_root_idc3(K|] }|k( ywrIr=r?r%needles r&rAzJSessionSearchMixin.search_sessions_by_id..score..s;u5F?;src3@K|]}|jywrI) startswithrPs r&rAzJSessionSearchMixin.search_sessions_by_id..score..sME,,V4Msrr)r rlr rL)rv normalizedrQs r&scorez7SessionSearchMixin.search_sessions_by_id..scores{.1#''$-2E2.FCGGTfLgLmkmHn-oust!'')uJu; ;;M*MM1 TST Tvs B B c$|d|dfS)Nrrr=)itemrVs r&z:SessionSearchMixin.search_sessions_by_id..stAwQUVWQX@Yr6)r0N) r!r list_sessions_richrlrr rrsrr) rrrrGrrHr candidatesrankedr@rrQrVs @@r&search_sessions_by_idz(SessionSearchMixin.search_sessions_by_ids +2$$&,,.!I,,7OSVW\_`W`bgSh'7d]c-e  UtCH~ U# U  *-3YZ"(%.13111s B$rpcv |jjd|dy#tj$rYywxYw)uTrue if an FTS5 virtual table is queryable ("no such table" and "vtable constructor failed" — missing tokenizer / mid-teardown — both count as not).zSELECT 1 FROM z LIMIT 0TF)r)r2rr)rrps r&_fts_table_existsz$SessionSearchMixin._fts_table_existss=  JJ  vX> ?$$  s "88cd|jDcgc]}|j|s|c}Scc}w)z3Queryable FTS tables (caller holds ``self._lock``).) _FTS_TABLESr_)rrs r&_present_fts_tablesz&SessionSearchMixin._present_fts_tabless)#//O43I3I#3NOOOs--c |jd}|j5|j|j|j d|j D]*} |jj d|d|d|dz }, ddd|S#tj$r!}tjd ||Yd}~gd}~wwxYw#1swY|SxYw) aMerge fragmented FTS5 segments into one per index (``'optimize'``). Pure maintenance: changes neither results nor ``snippet()`` output, only layout and speed; VACUUM then returns the freed pages. Returns the number optimized. A quarantined handle never issues ``'optimize'``: it rewrites index segments in place and would compound structural damage (or a split WAL generation) instead of leaving it diagnosable.rNwriterr7r8z) VALUES('optimize')rzFTS optimize failed for %s: %s) _raise_if_db_corruptr'_raise_if_db_replacedr)_reopen_after_close_lockedrbr2rrrr)r optimizedrrs r& optimize_ftszSessionSearchMixin.optimize_ftss !!# ZZ O  & & (zz!///@//1 OOJJ&&cU!C5@T'UVNI O O//ONN#CS#NNO Os6AC "'B C C )CC C  C  Cc |jd}t|j5}|stj d dddy|j 5|j |j|jd|jD]D} |jjd|d|d|jj|d z }F dddddd|S#tj$r;}|jjtj d ||Yd}~d}~wwxYw#1swYdxYw#1swY|SxYw) uRebuild FTS5 indexes from ``messages`` (``'rebuild'``) — the recovery for a corrupt index that rejects writes while reads succeed. Two processes rebuilding one state.db concurrently corrupted production DBs, so this admits through ``fts_rebuild_admission`` and FAILS CLOSED, returning 0 on deferral (callers treat 0 as "no progress" and use the stale-FTS breadcrumb path). Returns indexes rebuilt. Uses the FTS5 ``'rebuild'`` command, which rewrites the internal b-tree segments from the content rows. Unlike ``optimize_fts`` (which merges existing segments), ``rebuild`` discards and recreates the index data entirely — the more destructive of the two, so it is quarantined the same way. See #50502. A full structural rebuild must never run concurrently in two processes sharing one state.db — that interleaving has structurally corrupted the database in production (PR #93200) — so this admits through the cross-process ``fts_rebuild_admission`` authority and FAILS CLOSED: if another process holds the rebuild lock beyond the bounded wait, this call defers (returns 0) rather than racing it. Callers already treat 0 as "rebuild made no progress" and fall back to the stale-FTS breadcrumb path, which retries in-process from the gateway housekeeping tick (``retry_deferred_fts_recovery``) and at next startup. rz]Deferred in-place FTS rebuild: another process holds the rebuild authority for this state.db.Nrdrer7r8z) VALUES('rebuild')rzFTS rebuild failed for %s: %s)rfrdb_pathrrr'rgr)rhrbr2r*rrrollback)rrebuiltadmittedrrs r& rebuild_ftszSessionSearchMixin.rebuild_ftssH& !!# "4<< 0 RHsu  R R  R**,::%33G3D335RCR **\#auDW+XY ))+1  R R  R"#33R ++-'FSQQR R R  R"sUE  E AEAC/EE /D= 1D8 3E8D= =EE E  E) max_commandsrrqc td|| |j}td|d}|j5|jD]}|js"|j j d|d|dt|D]_}|j j}|j j d|d|d|f|d z }|j j|z d ks_d |_ddd|S#1swY|SxYw) a^Run bounded FTS5 ``'merge'`` commands against each present index. A positive merge rank stops after ~that many output pages, so each command holds the write lock for milliseconds regardless of index size (``'optimize'`` takes 9-18 s per index on a 10 GB DB). ``usermerge`` is lowered to its minimum of 2 (persisted in ``%_config``, once per instance) so a merge acts on ANY level with >= 2 segments; at the default 4 a fragmented index cannot converge. Up to *max_commands* per index, stopping on the no-progress signal ``total_changes`` delta < 2 (the INSERT itself is 1). Each command is its own implicit transaction, so processes interleave mid-pass. Missing tables are skipped (optimize_fts_storage drops + backfills them live); other SQLite errors propagate. Returns commands executed.rNrqrr7r8z, rank) VALUES('usermerge', 2)z, rank) VALUES('merge', ?)rrT) rt_FTS_MERGE_COMMANDS_PER_PASSr'rb_fts_usermerge_floor_appliedr)r2r total_changes)rrrqexecutedrr@befores r&rz+SessionSearchMixin._merge_fts_incrementallys k9-  <'R>$>@0& t  DOO@E   !&((S(T  c  d  $0 $LP`dM3&xc3h0@$0F'GHM3Y]M3 c3hM3dXY0E, , 25, ?B, QT, U38_-, JNsTWx., ^JO##&)#BF#SWX\]`be]eXfSg#N"!C"!C"!"!H4C4D449999     ? ?# ? ?AAAA kSkTkk -3-4- 33& c  3 RU _b %*39%5  jn7;3cYabeYf'/}JRSWX\]`be]eXfSgJh,BS B4BcBVYB^bcghkmphpcq^rB #B3#B5d3iRU9V3W#B#BJ u u$' u14 urs   III:   > *52::)")),?"@!ACD/0"fX-N+BJJz*eYi5 =Hhi  $ YY'++!6Y _c_d_999MCMDIM+s+tCy+ 73737>>C>D>  C  3  tCy  TW  dg  lo  "#t& 9&"&9=&NVW[\_W`Na&d3i(&7?S 7J&sm&08}&IM&