+ jHjoaR*t.0tRt^RIt^RIt^RIt^RIt^RIt^RIHt^RI H t ^RI H t ^RI HtHtHtHtHtHtHt^RIHtHt]P0!R4tRtR tR R ltR R ltRRltR+tRRlt RRlt!RRlt"RR,RR-/t#]^kRRlt$RRlt%RRlt&RRlt'R R!lt(R"R#lt)R$R%lt*R.t+R&R'lt,!R(R)4t-R#)/zFTS5 index setup for SessionDB: the CJK-bigram (cjk_unicode61) index DDL and tokenizer loader, per-table schema ensure with tokenizer-capability fallback, FTS-scoped corruption detection and the atomic fail-open trigger detach.NPath)Sequence)get_hermes_home)FTS_CJK_STALE_KEYFTS_SQL FTS_STALE_KEYFTS_TRIGRAM_SQL_FTS_CJK_TRIGGERS _FTS_TRIGGERSrouted_sessions_setting)is_fts_scoped_corruption_erroris_sqlite_lock_error hermes_statea\ CREATE VIEW IF NOT EXISTS messages_fts_cjk_src AS SELECT id, role, content, tool_name, tool_calls FROM messages WHERE role <> 'tool'; CREATE VIRTUAL TABLE IF NOT EXISTS messages_fts_cjk USING fts5( content, tool_name, tool_calls, content='messages_fts_cjk_src', content_rowid='id', tokenize='cjk_unicode61' ); a  CREATE TRIGGER IF NOT EXISTS messages_fts_cjk_insert AFTER INSERT ON messages WHEN new.role <> 'tool' AND (new.id > COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_high_water'), -1) OR new.id <= COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_progress'), -1)) BEGIN INSERT INTO messages_fts_cjk(rowid, content, tool_name, tool_calls) VALUES (new.id, new.content, new.tool_name, new.tool_calls); END; CREATE TRIGGER IF NOT EXISTS messages_fts_cjk_delete AFTER DELETE ON messages WHEN old.role <> 'tool' AND (old.id > COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_high_water'), -1) OR old.id <= COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_progress'), -1)) BEGIN INSERT INTO messages_fts_cjk(messages_fts_cjk, rowid, content, tool_name, tool_calls) VALUES ('delete', old.id, old.content, old.tool_name, old.tool_calls); END; CREATE TRIGGER IF NOT EXISTS messages_fts_cjk_update AFTER UPDATE OF content, tool_name, tool_calls, role ON messages WHEN (old.content IS NOT new.content OR old.tool_name IS NOT new.tool_name OR old.tool_calls IS NOT new.tool_calls OR old.role IS NOT new.role) AND (old.id > COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_high_water'), -1) OR old.id <= COALESCE((SELECT CAST(value AS INTEGER) FROM state_meta WHERE key = 'fts_cjk_rebuild_progress'), -1)) BEGIN INSERT INTO messages_fts_cjk(messages_fts_cjk, rowid, content, tool_name, tool_calls) SELECT 'delete', old.id, old.content, old.tool_name, old.tool_calls WHERE old.role <> 'tool'; INSERT INTO messages_fts_cjk(rowid, content, tool_name, tool_calls) SELECT new.id, new.content, new.tool_name, new.tool_calls WHERE new.role <> 'tool'; END; c$V^8dQhR\/#returnr)formats"5/home/ubuntu/.hermes/hermes-agent/hermes_state_fts.py __annotate__res[[$[c\P!R4pV'd\V4P4#\ 4R, R, #)z1Location of the cjk_unicode61 loadable extension.HERMES_FTS5_CJK_SOlibzlibfts5_cjk.so)osgetenvr expanduserr)envs rfts5_cjk_so_pathres9 ))( )C%(49   !Zo.?%.GJZ.ZZrc$V^8dQhR\/#rbool)rs"rrrksZZZrc\RR4pVRJ;'g+\V4P4P4R9#)zKconfig.yaml ``sessions.cjk_fts`` (default on) for the profile being served.cjk_ftsHERMES_CJK_FTSN)0falseoffno)r strstriplower)values r_cjk_fts_config_enabledr.ks= #I/? @E D= Y YCJ,,.446>YYYrcDV^8dQhR\PR\/#)rconnr)sqlite3 Connectionr")rs"rrrqs '"4"4rcf\4p\4'dVP4'gR#VPR4VP \ V44VPR4R# TPR4i;i \ d\PRTRR7R#i;i)zBest-effort load of the cjk_unicode61 tokenizer; False (never raises) when the .so is absent, ``sessions.cjk_fts`` is off, or loading is compiled out.FTz#fts5_cjk extension load failed (%s)exc_info) rr.existsenable_load_extensionload_extensionr* Exceptionloggerwarning)r0paths& rload_fts5_cjk_extensionr=qs  D " $ $DKKMM  ""4( .   D *  & &u -  & &u -  IF NOT EXISTS `` statement in *ddl* (``IF NOT EXISTS`` and ``;`` dropped), i.e. what ``sqlite_master.sql`` holds once the DDL has run.zCREATE z IF NOT EXISTS rD;z IF NOT EXISTSrE)indexrKreplace)rMrNrOstartends&&& r_expected_stored_sqlrVsP IIv_TF!< =E ))C C #../?QG HHrc<V^8dQhR\R\R\/#)rrMtablerrB)rs"rrrs!==c=#=#=rc\VRV4#)z VIRTUAL TABLE)rV)rMrXs&&r_expected_vtable_sqlrZs _e <>UWf hhphphr 59 :T D,s# #T :: :sAc<V^8dQhR\R\R\/#)rrXstoredrr*r")rs"rrrs!**C***rcnV\V,RR9dR#VR8XdRV9#RVP49#)zA definition an earlier release created. The two shape checks keep every store the pre-registry code treated as old (any base index without ``tool_name``, any trigram with ``tool_calls``) on the opt-in path.NTr> tool_name tool_calls)FTS_VTABLE_LAYOUTSr,)rXrgs&&r_is_older_layoutrns@#E*3B// &(( 6<<> ))rcDV^8dQhR\PR\/#r\r1r^r")rs"rrrs"]]'..]T]rcR\V4PR4\R,9#)zy``messages_fts`` holds a definition this code or an earlier release created; the open-time realign replaces nothing else.r>)regetrm)r]s&rbase_index_recognisedrss% 6 " & &~ 6:L^:\ \\rc0V^8dQhR\R\/#)rrgrrh)rs"rrrsFFFFrc&RV9;'dRV9#)zExternal-content base index not reading ``messages_fts_src``: the next read-write open rebuilds it from the current DDL (``_migrate_misaligned_fts_source``), so it is never opt-in work.rjmessages_fts_src)rgs&rbase_index_realigned_on_openrxs & E E%7v%EErcZV^8dQhR\PR\\,/#r\)r1r^listr*)rs"rrrs& X XW^^ XS Xrc\V4P4UUu.uF4wr\W4'gKVR8Xd\V4'dK2VNK6 upp#uuppi)aFTS tables on an older release's definition that only ``optimize-storage`` replaces; empty otherwise. Shared by the update notice, doctor and optimize-storage's demote decision. An absent table is not drift (the trigram index is optional; the base index is created on open), nor is a base index the next open realigns, nor a definition this code does not recognise. The cjk index owns its own markers. r>)reitemsrnrx)r]rXrgs& rfts_layout_driftr}s\(7v'>'D'D'F X'Fme. n,1Mf1U E'F XX XsAAAcDV^8dQhR\PR\/#r\rp)rs"rrrs" q qw~~ q$ qrcx\V4p\;QJd*RVP44F 'gK RM RM!RVP444'dR#RV9dR#VPR4P 4pVRJ;'d%\ V^,4\ \RR48H#) zTrue when every present FTS table, and the base index's source view, is exactly what this code creates: the only state the ``fts_storage_version`` stamp may be written over.c3T"TFwrV\V,R,8gxK R#5i)Nrl)rm).0rXrAs& r %fts_layout_current..s" Q.JE3$U+B/ /.s&(TFr>OSELECT sql FROM sqlite_master WHERE type = 'view' AND name = 'messages_fts_src'NVIEWrv)reanyr|rafetchonerKrVr)r]rgrows& rfts_layout_currentrsV $F s Q&,,. Qsss Q&,,. QQQV# ..j k t t vC d? p p|CF37KGU[]o7ppprcDV^8dQhR\PR\/#r\rp)rs"rrrs"qqgnnqqrcVPR4P4pVRJ;'d%\V^,4\\RR48g#)zv``messages_fts_src`` exists but differs from what this code creates (``CREATE VIEW IF NOT EXISTS`` never replaces it).rNrrv)rarrKrVrr]rs& rbase_source_view_is_stalersH ..j k t t vC d? p p|CF37KGU[]o7ppprc|V^8dQhR\PR\\,R\\,/#)rr]familiesr)r1r^rr*rz)rs"rrrs07>>Xc]W[\_W`rc.pVFpVPRV34P4pV'dK.\Uu.uF qS RV 2NK ppVPRRPRV44 R2V4P 4Uu.uF qw^,NK ppV'gKVFp VPRV R24K \ P R VR PV44VPV4K V#uupiuupi) aDrop, per family, shadow tables whose virtual table row is absent from sqlite_master. Matches exact shadow names only (never a prefix LIKE, so the base family cannot reach ``messages_fts_trigram_*``) and leaves a family alone whenever its vtable is live. The shadows are derived index state; the caller recreates and rebuilds from ``messages``. Returns the families that were repaired. zbSELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ? AND sql LIKE 'CREATE VIRTUAL TABLE%'_zASELECT name FROM sqlite_master WHERE type = 'table' AND name IN (,c3&"TFpRxK R#5i?Nrwrrs& rr1_drop_orphan_fts_shadow_tables.. sXnfmabY\fm)z DROP TABLE ""zRDropped orphan FTS5 shadow tables of %s (%s); the index is recreated from messagesz, )rar_FTS5_SHADOW_SUFFIXESrHrcr:r;append) r]rrepairedfamily vtable_livesuffixshadowsrorphansrOs && r_drop_orphan_fts_shadow_tablesrsHnn 3 I  (*   6KL6KFXQvh'6KL%+^^OPSPXPXXnfmXnPnOoop q &  (*&&cq66& D NN\$q1 2 ` DIIg&  +, OMs DDcta]tRtRtoRt]V3RlRl4t]V3RlRl4t]V3RlRl4t]V3R lR l4t ] V3R lR l4t V3R lRlt V3RlRlt V3RlRlt]V3RlRl4tV3RlRlt]V3RlRl4tRRRR/V3RlRlltRtRtR tR!tV3R"lR#ltR%tR$tVtR#)&SessionFtsSetupMixinizOFTS table/trigger lifecycle shared by schema init, optimize and the write path.c:<V^8dQhRS[PRS[/#rexcrr1OperationalErrorr")r __classdict__s"rr!SessionFtsSetupMixin.__annotate__s%vv(@(@vTvrc \V4P4pRV9;'dRV9;'g\PV4#)zONo FTS5 module, or an optional tokenizer missing (same capability-error shape).zno such modulefts5)r*r,r_is_trigram_unavailable_errorrerrs& r_is_fts5_unavailable_error/SessionFtsSetupMixin._is_fts5_unavailable_errors@#hnn C'99FcMuu>R>p>pqt>uurc:<V^8dQhRS[PRS[/#rr)rrs"rrr"s%``7+C+C``rc X\V4P4pRV9;'gRV9#)zOnly an optional tokenizer is missing (trigram needs SQLite >= 3.34; cjk_unicode61 is loadable): "this one index can't be served", never "disable FTS".zno such tokenizer: trigramz no such tokenizer: cjk_unicode61)r*r,rs& rr2SessionFtsSetupMixin._is_trigram_unavailable_error!s.#hnn+s2__6X\_6__rc:<V^8dQhRS[PRS[/#r\rp)rrs"rrr)s# E E'.. ET Erc VPR4P4pVRJ;'dRV^,;'gR9#)u]messages_fts exists in ANY pre-v23 shape: every legacy shape lacks tool_name, so "stored CREATE lacks tool_name" catches them all. False when absent. v23's messages_fts is external-content over THREE real columns (content, tool_name, tool_calls). Every pre-v23 shape lacks the tool_name/tool_calls columns — whether the old inline single-column form (v11..v22) or the even older external-content single-column form (v10-era, pre-#16751). We therefore detect "needs optimize" as "the stored CREATE lacks the tool_name column", which is the precise v23 marker and correctly catches BOTH legacy variants. zLSELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'messages_fts'NrjrE)rarrs& r_db_has_legacy_inline_fts.SessionFtsSetupMixin._db_has_legacy_inline_fts(sBnn Z (* $DD;s1v||#DDrc:<V^8dQhRS[PRS[/#r\rp)rrs"rrr9s#JJgnnJJrc VPR4P4pVRJ;'d%RV^,;'gRP49#)z[True when the trigram vtable still includes the tool_calls payload (FTS_STORAGE_VERSION 1).zTSELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'messages_fts_trigram'NrkrE)rarr,rs& r%_db_has_trigram_tool_calls_projection:SessionFtsSetupMixin._db_has_trigram_tool_calls_projection8sKnn b (* $II>.d.rc *\\V44#)zdTrue when optimize-storage has layout work: a stored FTS vtable differs from what this code creates.)r"r})clsr]s&&r_db_needs_fts_storage_upgrade2SessionFtsSetupMixin._db_needs_fts_storage_upgrade@s$V,--rc8<V^8dQhRS[PRR/#rrrNr1r)rrs"rrrEs!   W-E-E  $  rc \VRR4'dR#RVn\PRVP\ P V4R#)zHLog once that the trigram tokenizer is missing; base FTS5 stays enabled._trigram_unavailable_warnedFNTzSQLite trigram tokenizer unavailable for %s (requires SQLite >= 3.34, this build is %s); CJK/substring search will fall back to LIKE: %s)getattrrr:infodb_pathr1sqlite_versionselfrs&&r_warn_trigram_unavailable.SessionFtsSetupMixin._warn_trigram_unavailableEsC 46 > > +/(  > LL  " "   rc8<V^8dQhRS[PRR/#rr)rrs"rrrSs!   '*B*B  t  rcRVnVP'dR#RVn\PRVPV4R#)FNTzSQLite FTS5 unavailable for %s; full-text session search disabled. Run `hermes update` to rebuild the venv with a current Python (managed uv guarantees FTS5). (underlying error: %s)) _fts_enabled_fts_unavailable_warnedr:r;rrs&&r_warn_fts5_unavailable+SessionFtsSetupMixin._warn_fts5_unavailableSs?!  ' ' ' '+$ R LL   rc<V^8dQhRR/#)rrNrw)rrs"rrr`sK,K,K,rc >\VPR4P44pVP'gV'dVPRRP R\ 44 R2\ 4P 4Uu.uF q3^,NK ppV'dS\PR\44VPR\34VFpVPRV 24K R Vn R #TP\4T'gTPR\34TPR4P4^,^8dRTPR4P4^,pR\!T43R3FwrxTPRYx34K TPR\34P4'd R Tn R #TP\"4TPR4P4p T '*Tn R #uupi \Pd#\PR R R 7R Tn R #i;i \Pd#\PRR R 7R Tn R #i;i)uCreate / repair / self-heal the CJK-bigram index (see the module comment). Sets ``_fts_cjk_available``; never raises. Loaded + absent → create (a populated DB gets backfill markers and is NOT served until optimize-storage backfills); loaded + present → ensure triggers, honour the stale breadcrumb; NOT loaded + live triggers → drop them (INSERTs must not fail at trigger time).zNSELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'messages_fts_cjk'zCSELECT name FROM sqlite_master WHERE type = 'trigger' AND name IN (rc3&"TFpRxK R#5irrwrs& rr>SessionFtsSetupMixin._ensure_fts_cjk_schema..ns0P>O>Orru messages_fts_cjk triggers present but the cjk_unicode61 tokenizer is unavailable (%s) — dropping the cjk triggers so message writes keep working. CJK search falls back to trigram/LIKE; run `hermes sessions optimize-storage` on a host with the extension to rebuild.z^INSERT INTO state_meta (key, value) VALUES (?, '1') ON CONFLICT(key) DO UPDATE SET value = '1'DROP TRIGGER IF EXISTS FNzHmessages_fts_cjk presence check failed; CJK search stays on trigram/LIKETr4z$DELETE FROM state_meta WHERE key = ?z2SELECT COUNT(*) FROM messages WHERE role <> 'tool'z)SELECT COALESCE(MAX(id), 0) FROM messagesfts_cjk_rebuild_high_waterzgINSERT INTO state_meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.valuez&SELECT 1 FROM state_meta WHERE key = ?zISELECT 1 FROM state_meta WHERE key = 'fts_cjk_rebuild_high_water' LIMIT 1z@messages_fts_cjk ensure failed; CJK search stays on trigram/LIKE)fts_cjk_rebuild_progressr&)r"rar_fts_cjk_loadedrHr rcr:r;rr_fts_cjk_availabler1r executescriptFTS_CJK_TABLE_SQLr*FTS_CJK_TRIGGER_SQL) rr] cjk_presentrlivetrighwkvbackfill_pendings && r_ensure_fts_cjk_schema+SessionFtsSetupMixin._ensure_fts_cjk_schema`s % v~~` hjK'''*0..((+0P>O0P(P'QQRT)+hj +!"+!QaDD+!D" = -.I.0 %)D"NN-DTF+KL%)*/'5(D ,  !2 3EHYG[\>>"VW``bcdehii(ST]]_`abB5s2w?Ab!TF!~~FIZH\]ffhh*/'  !4 5%~~[ hj +;&:D #s"2''  NN)-  ',D #   B'' , NN)-  ',D #  ,sRAH+AH+H&A#H+I%B=I%I%$AI%&H++3I"!I"%3JJc8<V^8dQhRS[PRR/#)rr]rN)r1r^)rrs"rrrs7>>drc\FpVPRV 24K R# \PdK6i;i)rN)r rar1r)r]triggers& r_drop_fts_triggers'SessionFtsSetupMixin._drop_fts_triggerss?$G !8 BC%++  s %>>cF<V^8dQhRS[PRS[RS[RS[/#)rr] table_namerMr)r1r^r*r")rrs"rrrs.SsW[rcBVPW4pVfR#VPV4R# \Pd^pTP T4'ghTP T4'dTP T4MTPT4Rp?R#Rp?ii;i)NFT)_fts_table_proberr1rrrrr)rr]rrMstatusrs&&&& r_ensure_fts_schema'SessionFtsSetupMixin._ensure_fts_schemas&&v: >    %'' 2237711#66..s3++C0 s,B.B0#BBc:<V^8dQhRS[PRS[/#r)r1 DatabaseErrorr")rrs"rrrs!33G,A,A3d3rc \V4#)aNCorruption SQLite identifies as FTS-scoped (SQLITE_CORRUPT_VTAB, or an ``fts5:`` report naming ``messages_fts*`` on builds without result codes); a bare malformed image is structural. One rule, shared with ``classify_persistence_error`` and the gateway transcript retry: see :func:`hermes_state_errors.is_fts_scoped_corruption_error`.)r )rs&r_is_fts_write_corruption_error3SessionFtsSetupMixin._is_fts_write_corruption_errors .c22rdeadlineN patience_scb<V^8dQhRS[PRS[R,RS[R,RS[/#)rrrNrr)r1rfloatr")rrs"rrrsCHH((H7.s4TBSQSBSrz ) LIMIT 1zNCould not detach corrupt FTS indexes; canonical write still cannot proceed: %szstate.db FTS indexes remain corrupt (%s); disabled FTS sync and retrying the canonical write. Search temporarily uses LIKE until a later SessionDB open rebuilds the indexes.) rr_WRITE_PATIENCE_Stime monotonic_raise_if_db_corrupt_lock_raise_if_db_replaced_conn_reopen_after_close_lockedrarrHr rr_drop_all_fts_triggersr]commit BaseExceptionrollbackr1Error isinstancerr_sleep_before_write_retryr:error _fts_stale_trigram_availabler)rrrrcjk_triggers_present detach_excs&&$$ r_enter_fts_fail_open)SessionFtsSetupMixin._enter_fts_fail_opens:   (K(KC(P(P  //J  ~~'*4H  % %d % 3( ZZZ..0zz)777HJJ&&'89 **T*, 04zz/A/A,,/HH4TBS4T,T+UV&&. 0 #(* - 0 JJ..!X!2 4 33DJJ4E4E4GH ))+1 8!"'"'  ;   3) ++-3 Z:== z7+C+CDDI]^hIiIi66xLL d sR0H%A HA4G(A$G(+H%('HH H" H%%J*:AJ% J%%J*ig@g?fts_v22_trash_c <V^8dQhRS[/#rr!)rrs"rrr(sdrc  \VPRVPPRR4R,34P 44#)zTrue when demoted v22 shadow tables are still awaiting teardown. Caller must hold ``self._lock`` (or pass a migration-time cursor).zSSELECT 1 FROM sqlite_master WHERE type = 'table' AND name LIKE ? ESCAPE '\' LIMIT 1rz\_%)r"ra_FTS_TRASH_PREFIXrSr)rr0s&&r_has_fts_trash#SessionFtsSetupMixin._has_fts_trash(sFDLL b  # # + +C 7# = ?  (* r)rrrrrr)r>r?messages_fts_cjk)__name__ __module__ __qualname____firstlineno____doc__ staticmethodrrrr classmethodrrrrrrrr_FTS_REBUILD_CHUNK_ROWS_FTS_REBUILD_DUTY_FACTOR_FTS_REBUILD_MIN_PAUSEr#r$ _FTS_TABLES__static_attributes____classdictcell__)rs@rrrsYvv ``  E EJJ..      K,K,Z$33HFJHgkHH`""  )OKrrc|V^8dQh/^\9d(\\\\R3,3,;R&#)r.rm)__conditional_annotations__r_r*tuple)rs"rrrs9LL|DeCHo-.}Lr)r>r?)zWCREATE VIRTUAL TABLE messages_fts USING fts5(content,content=messages,content_rowid=id)z5CREATE VIRTUAL TABLE messages_fts USING fts5(content)zpCREATE VIRTUAL TABLE messages_fts USING fts5(content,tool_name,tool_calls,content='messages',content_rowid='id')zxCREATE VIRTUAL TABLE messages_fts USING fts5(content,tool_name,tool_calls,content='messages_fts_src',content_rowid='id'))zrCREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content,content=messages,content_rowid=id,tokenize='trigram')zPCREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content,tokenize='trigram')zCREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content,tool_name,tool_calls,content='messages_fts_trigram_src',content_rowid='id',tokenize='trigram')zCREATE VIRTUAL TABLE messages_fts_trigram USING fts5(content,tool_name,content='messages_fts_trigram_src',content_rowid='id',tokenize='trigram'))contentdatadocsizeidxconfig)/r5r+loggingrrFr1r pathlibrtypingrhermes_constantsrhermes_state_commonrrrr r r r hermes_state_errorsr r getLoggerr:rrrr.r=rbrKrVrZrmrernrsrxr}rrrrrr)r5s@rrCsL  ,,,,T   > *, )X[ Z $;YI=2*; *] F X qqHDZOZOr