߲j}dZddlmZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlmZddlmZmZmZmZmZddlmZddlmZddlmZdd lmZmZm Z m!Z!m"Z"ejFd Z$d Z%d Z&d Z'd Z(dZ)dZ*dZ+dZ,dZ-dZ.dZ/dfdZ0dgdZ1dhdZ2didZ3djdZ4dkdZ5dddldZ6dmdZ7ejpdfdZ9dnd Z:e6Z;dod!Zdrd$Z?dsd%Z@dsd&ZAdtd'ZBdud(ZCdvd)ZDdwd*ZEdwd+ZFdxd,ZGdhd-ZHdyd.ZIdd/dzd0ZJd{d1ZKe(fd|d2ZLd}d3ZMd~d4ZNd5d6dd7ZOd8d9dd:ZPd8d9dd;ZQdd<ZRdd=ZSdod>ZTdd?ZUejpdfd@ZVdddA ddBZWdudCZXe jdDZZe jdEZ[e jdFZ\ddGZ]dhdHZ^dsdIZ_dhdJZ`dddKZadLdMddNZbdsdOZcddPddQZdddR ddSZeddTZfdodUZgdodVZhdodWZidodXZjdodYZkdZehd[d\fd]eid^d_fd`ejdadbfdcekdddffZlddeZmy)zstate.db repair, backup and writability preflight (split from hermes_state). Patchable helpers are looked up as module globals at call time, so tests patch ``hermes_state_repair.``. ) annotationsN)Path)AnyDictListOptionalTuple)get_hermes_home)report_startup_progress)read_only_db_uri)_acquire_db_flock_clear_lock_holder_record_describe_lock_holder_read_lock_holder_recordis_advisory_lock_contention hermes_stateg?i)-wal-shmz-journali)))\`ig{Gz?) messages_ftsmessages_fts_trigrammessages_fts_cjkzyFree disk space, then retry (or recover manually with `hermes sessions recover --source {db_path} --inspect-only` first).c"fdtDS)z>The three SQLite sidecar paths for *db_path* (present or not).c3ZK|]"}jj|z$ywN with_namename).0suffixdb_paths 8/home/ubuntu/.hermes/hermes-agent/hermes_state_repair.py z_sidecars..Js% XG  gllV3 4 Xs(+)_DB_SIDECAR_SUFFIXESr%s`r& _sidecarsr*Hs XCW XXc ddlm}m} |||5|cdddS#t$r dt}}Y0wxYw#1swYyxYw#|tf$rYywxYw)aRun *reader()* under ``hermes_cli.sqlite_safe_read.offline_file_access``. ``close()`` on ANY raw descriptor cancels every POSIX advisory lock this process holds on the file, including a peer connection's RESERVED lock (``sqlite_safe_read`` rule 1), so a raw read is only safe with no live connection; ``None`` when that makes it unsafe or the file is unreadable. Scaffold/embed installs without hermes_cli have no tracked connections.r)LiveConnectionErroroffline_file_accessc*tjSr) contextlib nullcontext)_p_ks r&z_read_offline..WsZE[E[E]r+)whatN)hermes_cli.sqlite_safe_readr-r. ImportErrorOSError)r%r5readerr-r.s r& _read_offliner:MsnhX t 4 8   h4]`g0h    )s7& A > A ;;AA A AAcddlm}m}|5t||vr dddy|j t| dddy#1swYyxYw)zClaim the one-shot per-process repair attempt for *db_path*: True for the first caller, False afterwards (bounds the repair/reopen loop and stops concurrent callers racing surgery on one file).r)_repair_attempt_lock_repair_attempted_pathsNFT)rr<r=stradd)r%r<r=s r&_claim_repair_attemptr@_sQK  w<2 2 ##CL1 sAAAc|j|j|z} |jjdd||j dfS#t $r+}t jd|d||||dfcYd}~Sd}~wwxYw)zDOpen ``.`` for locking; on failure warn and return None.T)parentsexist_okza+bzCould not open state.db u lock %s (%s) — N)r!r"parentmkdiropenr8loggerwarning)r%r$r5tail lock_pathexcs r&_open_lock_filerLjs!!',,"78Itd;)../// 1$7I$PR[]`a$s/A B A?9B?Bcddl}|jd|j|jt ||dy)Nr)msvcrtseeklockingfilenogetattr)handle flag_namerOs r& _msvcrt_lockrVus- KKN NN6==?GFI$>Br+cddlm}|r t|dyddl}|j |j |j |jzy)zJTake the advisory lock on *handle* without waiting (raises on contention).r _IS_WINDOWSLK_NBLCKN)rrYrVfcntlflockrRLOCK_EXLOCK_NB)rTrYr[s r&_try_lock_nonblockingr_{s6(VZ( FMMOU]]U]]%BCr+F clear_recordcTddlm}tj|5tjt 5|r t |dn;ddl}|r t||j|j|jddddddy#1swYxYw#1swYyxYw)z>Drop the advisory lock on *handle* (best effort) and close it.rrXLK_UNLCKN) rrYr0closingsuppressr8rVr[rr\rRLOCK_UN)rTrarYr[s r&_release_lock_handlergs}(   F #8Z%8%8%A8   , )&1 KK  7888888s#BA BBB BB'c>tj|z} t|y#ttf$rf}t |st jd||Yd}~ytj|k\rYd}~ytjtYd}~nd}~wwxYw)zeWindows counterpart of ``_acquire_db_flock``: True / False (timed out) / None (non-contention error).TueCould not acquire state.db repair lock %s (%s) — skipping schema surgery on a non-contention error.NF) time monotonicr_BlockingIOErrorr8rrGrHsleep_REPAIR_LOCK_POLL_SECONDS)rJrTtimeoutdeadlinerKs r&_acquire_repair_lock_windowsrps~~')H  2 !& )) 2.s3 78A3H~~8+ JJ0 1 1 2 s &B"BB8BBc #Kddlm}m}t|ddd\}}|dyd} |rt |||}nt t |||td\}}|d}n0|s.tjd ||t|rdn t|||rt|d y|jy#|rt|d w|jwxYww) aSerialize state.db schema surgery across processes. Yields True when this process holds the repair lock, False when the bounded acquire timed out or the lock file could not be opened; on False the caller must NOT do surgery (unlocked surgery IS the interleaving this prevents). ``flock``: the kernel drops it when the holder dies (a pidfile would wedge every future repair); a forked child inheriting the fd is the exception, so the acquire records pid + start time and breaks a provably dead holder's lock. Bounded because a live repairer can sit in ``VACUUM`` for minutes. An unopenable lock file (no space/inodes/descriptors) fails closed too: a sibling that opened ITS handle before the disk filled may be inside surgery. The acquire is still bounded because a *live* repairer can legitimately sit in ``VACUUM`` for minutes on a large DB, and an unbounded wait would hang the caller's open with no traceback (the failure shape of #36644). r)rY_REPAIR_LOCK_TIMEOUT_SECONDSz .repair.lockrepairzOskipping schema surgery rather than running it without cross-process authority.NFzstate.db repair lockustate.db repair lock %s held by another process for more than %.0fs — skipping schema surgery in this process to avoid racing the repairer. Recorded holder: %s.Tr`)rrYrrrLrpr r>rmrGrHrrrgclose)r%rYrrrJrTacquireds r&_cross_process_repair_lockrvs G'+|~Iv ~ H 3IvGcdH0YIe1JLb d Hf  H NNh$&B0JbciJjk m  d ; LLN  d ; LLNs#CA(B/!C/"CCct|ddd\}} | t||S#ttf$r|j YywxYw)zNon-blocking cross-process lock for one auto-maintenance pass (None = skip the pass: otherwise two startups both pass the interval check and the second prunes a row the first has only just closed recoverably).z.auto-maintenance.lockzauto-maintenancezskipping automatic maintenance.N)rLr_rkr8rt)r% _lock_pathrTs r&"_try_acquire_auto_maintenance_lockrys[)2JL^aBCJ   !& ) W % s"AAc |jdjd}|jdt|dzdzy#tjt t f$r }tjd|Yd}~yd}~wwxYw)amIncrement the schema cookie after direct ``sqlite_master`` surgery. Ordinary DDL bumps it and peers compare it before running a prepared statement (how they discard a cached schema); ``writable_schema=ON`` edits do NOT, so live connections would keep firing triggers into ``messages_fts*`` shadow tables that no longer exist. Best-effort, never raises.zPRAGMA schema_versionrzPRAGMA schema_version=rNiz)Could not bump state.db schema cookie: %sN) executefetchoneintsqlite3 DatabaseError TypeError IndexErrorrGrH)conncurrentrKs r&_bump_schema_cookiersy I,,67@@B1E -s7|a/?:.M-NOP  ! !9j 9IBCHHIsAAB&BBct|dkr|St|}tD]\}}d||z z|||t|S)zJZero the commit-counter fields so ordinary writes don't re-key the ledger.r)len bytearray#_FINGERPRINT_VOLATILE_HEADER_RANGESbytes)headbufstartends r&_mask_volatile_headerrsL 4y2~ D/C91 s C%K0E#1 :r+cBttt|tzS)zLFree space required *beyond* the copy itself, for a volume of *total_bytes*.)max_REPAIR_BACKUP_MIN_FREE_BYTESr}_REPAIR_BACKUP_FREE_FRACTION) total_bytess r&_repair_backup_headroom_bytesrs ,c+@\2\.] ^^r+cT |jjtdt|Dz}t j |j }d||jt|jfS#t$r$}d|j d|d|ddddfcYd}~Sd}~wwxYw)a+``(error, bundle_bytes, free_bytes, headroom_bytes)`` for *db_path*'s volume (main file plus every PRESENT sidecar); *error* is set (and the sizes zero) on stat()/disk_usage() failure. Fails CLOSED: the nearly-full volume these guards exist for is exactly where they are most likely to fail.c3rK|]/}|js|jj1ywr)existsstatst_sizer#ps r&r'z_disk_budget..s(+g\]\d\d\fAFFH,<,<+gs77z"could not determine free space on z (z); refusing the z$ rather than risk filling the volumerN) rrsumr*shutil disk_usagerDr8freertotal)r%refusalneedusagerKs r& _disk_budgetrs/||~%%+giPWFX+g(gg!!'..1 uzz#@#M MM /4W^^4DBseK[\c[de%%'(!Q/ //sAA:: B'B"B'"B'c t|d\}}}}||S||d|zz|zk\ryd|dz dd|jd|dz dd d|zdz dd |dz dd S) zEReturn an error unless snapshot, VACUUM and promotion can fit safely.zrepair snapshotNonly eA.2f GB free on z"; the repair snapshot needs up to zGB, VACUUM may need another zGB, and z8GB must remain as headroom. Free disk space, then retry.)rrD)r%errorsnapshot_bytesrheadrooms r&_repair_scratch_space_errorr s,8BS,T)E>4   ~^!34x??D3Js#;w~~.>>`#C((Da.FX\_E_`cDddl#~c""Z \]r+c tj|}t|d\}}}}||d|S||z |k\ryd|dz dd|jd |dz dd |dz dd | S) zDisk guard for the forensic copy: reason to refuse, or None. A full raw copy on a nearly-full volume (which a preceding repair loop may itself have caused) can finish off the disk and every process on the machine.r)z forensic copyNz. rrrrz; copying the damaged DB needs zGB and must leave z GB headroom. )_MANUAL_RECOVER_HINTformatrrD)r%hintrrrrs r&_backup_free_space_errorrs & &w & 7D".w"HE4x 4&!! d{hD3Js#;w~~.>>]^beh^hil]mn^C( dV =>r+cddlm}d}|gt|D]A}tjt 5||j jz }dddCt||tz S#1swY`xYw)zBound one SQLite snapshot by source size incl. sidecars (a WAL can hold committed rows not yet in the main file), so a healthy large-database copy is not cut off by the repair-lock timeout.r)rrN) rrrr*r0reFileNotFoundErrorrrr0_REPAIR_SNAPSHOT_MIN_THROUGHPUT_BYTES_PER_SECOND) source_pathrr source_bytes candidates r& _repair_snapshot_timeout_secondsr$s~:L!;Ik$:;5  !2 3 5 INN,44 4L 5 55 +\lowerr#mrKs r&r'z3_repair_failure_consumes_attempt..;s!mqCHNN$$m%()zfile is not a databasez database disk image is malformed) isinstancer~rrSr}SQLITE_CORRUPT SQLITE_NOTADBany)rK error_codes` r& _repair_failure_consumes_attemptr/sf c700 1-? FF*LT!w'='=w?T?T&UUU m.lm mmr+c>|j|jdzS)Nz.repair-attempts.jsonr r)s r&_repair_ledger_pathr>s   W\\,CC DDr+c*dfd }td|S)a Cheap identity for a damaged DB file: size + a bounded content sample. EXCLUDES mtime: a malformed-schema DB still accepts writes, so live writers, checkpoints and the strategies move mtime between passes; keyed on mtime every pass looked like a NEW file and the attempt counter reset forever. Hashing a multi-GB file per open is the cost this ledger avoids, so sample the head/tail slices any real repair, truncation or restore must change. ``None`` = identity unavailable (:func:`_read_offline`; a live peer is expected here since this runs BEFORE ``_backup_db_file``'s live-connection guard). Callers MUST NOT substitute a differently-shaped key: the ledger compares for equality, so alternating shapes never match and the unbounded loop returns.cj}td5}|jt}|j t d|j tz |j tkDr|jtnd}ddd|j dtjtzjddS#1swYJxYw)Nrbrr+: ) rrFread_FINGERPRINT_SAMPLE_BYTESrPrrhashlibsha256r hexdigest)stfhrrIr%s r&_samplez _db_fingerprint.._sampleMs \\^ '4  iB7745D GGC2::(AAB C9;F_9_27745ehD i**Qw~~.CD.ID.PQ[[]^a_abcdd  i is A,CC fingerprintreturnr>r:)r%rs` r&_db_fingerprintrBse - 99r+c*dfd }td|S)uRecovery-image identity for forensic-backup dedupe: whole file + sidecars. A DIFFERENT relation from :func:`_db_fingerprint` (never conflate them): the fingerprint answers "same repair epoch?" from head+tail only, and a live writer can commit into an *interior* page while preserving size and both 64 KiB slices — reusing a backup on that basis hands the operator a snapshot predating real user data. A forensic copy must claim byte identity, so this digests the ENTIRE main file plus every present sidecar (the WAL can hold uncheckpointed frames). ``None`` when a live connection makes the read unsafe (:func:`_read_offline`) — the caller then takes a fresh backup, never a false reuse. See #87409. ctj}dfgdttt D}|D]~\}}|j d|d|j jdjt|d5tfddD]}|j | ddd|jS#1swYxYw)Nmainc3LK|]\}}|js||fywr)r)r#sfxrs r&r'z<_backup_content_identity.._digest..es$'{VS!pqpxpxpza'{s$ $rrc&jdS)Ni)r)rsr&r4z;_backup_content_identity.._digest..ks"''+*>r+r+) rrzipr(r*updaterrencoderFiterr)hashermemberslabelpathchunkrr%s @r&_digestz)_backup_content_identity.._digestcs!G$|'{sCWYbcjYk?l'{|" )KE4 MMBugQtyy{':':&;2>EEG HdD! )R!">D)EMM%() ) )  )!! ) )s &CC zbackup-identityrr)r%rs` r&_backup_content_identityrWs " "3W ==r+ctjtt5t j t |jd}t|tr|nicdddS#1swYiSxYw)Nutf-8encoding) r0rer8 ValueErrorjsonloadsr read_textrdict)r%raws r&_read_repair_ledgerrqs^   Wj 14jj,W5???QR d+s444 Is AA++A5cFt|}|jd}t|} |||k(n=t|txr,|j |j jd}|xr"t|jddtk\S#t$rYywxYw)aWhether *db_path* has already burned its cross-restart repair budget. True only when the ledger records ``_MAX_PERSISTENT_REPAIR_ATTEMPTS`` failures against the CURRENT fingerprint. Never raises; a missing/corrupt ledger or unstatable DB reads as "not exhausted" (the in-process claim and cross-process lock still bound one run). Fingerprint unavailable (live connection) -> compare the SIZE the ledger recorded, otherwise a peer connection hides an exhausted budget on every pass.rrFfailed_attemptsr) rgetrrr> startswithrrr8r}_MAX_PERSISTENT_REPAIR_ATTEMPTS)r%ledgerrecordedfpsames r&%_persistent_repair_attempts_exhaustedrxs! )Fzz-(H  !B!#x2~ x % [(*=*=AWAW@XXY>Z*[   \C #4a89=\\\ sAB B B ctddlm}|}dtd|d|d|d|dt|jd S) zThe stable operator-facing diagnostic for an exhausted repair budget. The ``hermes`` commands carry the profile selector: a bare ``hermes`` follows ``active_profile`` (#105887).r)profile_cli_selectorz$automatic repair has already failed u times on this exact file — the corruption is beyond the schema/FTS repair strategies (likely b-tree page damage). Manual recovery required: restore a backup, or salvage with `hermes zsessions recover --source z; --inspect-only`, then (if it reports recoverable) `hermes z --output recovered-state.db` (recovery snapshots the damaged file first, then runs the page-level `.recover` lane on the copy; do NOT point a raw `sqlite3` shell at the live database). Delete z$ to force another automatic attempt.)hermes_constantsrrrr")r%r profile_args r&"_persistent_repair_exhausted_errorr sj6&(K23R2STCCN-OijqirsIIT Uopwoxy*'27788\  ^_r+)rct|} |r|jdyt|}|jd}||n t |}|t |t sy|}||k(rt|jdddznd}tjjjd }|jtj|||d d y#t$r } tj!d | Yd} ~ yd} ~ wwxYw)uZUpdate the persistent attempt ledger after a repair pass. Never raises. Keys on the post-attempt fingerprint (what the NEXT exhaustion probe sees). If a live connection makes it unavailable, keep the recorded key and still increment — dropping the pass lets a peer reset the budget every time. Never write a differently shaped key.T missing_okNrrrrNseconds)timespec)rr last_attemptrrz+Could not update state.db repair ledger: %s)runlinkrrrrr>r}datetimenow isoformat write_textrdumps ExceptionrGrH) r%repairedr ledger_pathrrrattemptsstamprKs r&_record_repair_outcomers &g.KK    $  / $W-::m,'3[9Q :h,B@HB3vzz"3Q781.s qvvr+T)keyreverse)r"rDiterdirrendswithr(r8sorted)r%prefixrfounds r&_existing_malformed_backupsr's ~/ 0F#NN224Aq8I8I&8QZ[Z`Z`ZiZij~ZAA %-t <<A  s.B BB.B2BB BBct||dD])}|gt|D]} |jd+y#t$r!}tj d||Yd}~@d}~wwxYw)zADelete all but the *keep* newest forensic backups (and sidecars).NTr z&Could not prune stale DB backup %s: %s)r'r*rr8rGrH)r%keepstalevictimrKs r&_prune_malformed_backupsr,sz,W5de<V0y/0 VF V  . VV VGQTUU Vs< A&A!!A&c |||f}ttt|Dcgc]R\}}|jr=||j |j |z|j |j |zfT}}}g} |g|D]\}} } t j|| g||D]-\} } } tj| | |j| /ycc}}w#t$rZgdg||D|D]B} tjt5| jdddd8#1swYAxYwwxYw)uwCopy DB + sidecars to *staging* names, then rename each into place. ORDER MATTERS: the main DB name is the bundle's commit marker (what ``_existing_malformed_backups`` counts), so sidecars publish FIRST and the main DB LAST — a failure partway never leaves a countable backup over a missing sidecar. On failure, staging files AND anything promoted are removed.c3(K|] \}}}| ywr)r#_sstaged_ds r&r'z)_publish_backup_bundle..sHNBHsTr N)rr(r*rr!r"rcopy2osreplaceappendrr0rer8r)r%staging backup_pathrr$sidecarsidecars publishedsrcr1_dst_srcdstr+s r&_publish_backup_bundler@si Wk *D'*+?7AS'Tj#FGX_XfXfXh'++GLL6,ABKDYDYZeZjZjmsZsDtujHj I "&!2!2 & C LLf % &!28!2T!2 " D&# JJvs #   S ! " j UH6G6G$6GHU9U /F$$W- /  . / / / /  s+AC?AC""8ED6- E6D? ; EcJtjt5ddlm}||r*d|d}t j d||d|fcdddS dddtjjjd}|j|jd|}tjd D]7}|jsn%|j|jd|d |}9 |jd |jd fD]V}|jj!|D]6}tjt"5|j%d ddd8Xtjt"5t'|dd x}rGt)|} | :t)|d| k(r)t j+d|d|ddfcdddSdddt-|x}t j d||d|fSt/||j|jd||t1||dfS#1swYxYw#1swY-xYw#1swYxYw#t2$r(} t j5d|| dd| fcYd} ~ Sd} ~ wwxYw)uRaw-copy a (possibly malformed) DB plus sidecars to a timestamped backup. Raw bytes on purpose: the DB won't open cleanly, so preserve them exactly for forensics. Returns ``(backup_path, None)`` or ``(None, reason)``; repair treats a refused backup as a HARD STOP because the bundle is the recovery path when every strategy fails. Refuses while a connection to this DB is live in the process: the raw read would ``close()`` a descriptor and cancel that connection's POSIX advisory locks (``hermes_cli.sqlite_safe_read``) — real case: one SessionDB enters repair while the gateway holds others. Dedupe: reuse the newest backup when byte-identical to the current recovery image (``_backup_content_identity`` — NOT mtime, NOT ``_db_fingerprint``); a repair loop once re-copied the same bytes on every restart. Staging names live OUTSIDE the ``.malformed-backup-`` prefix: inside it they count as a backup, sort NEWEST (prune kept partials, deleted intact copies) and dedupe could return one with no real forensic copy on disk. Refusal reasons (``_backup_free_space_error`` / ``_MANUAL_RECOVER_HINT``) point operators at the safe lane, `hermes sessions recover --source --inspect-only`, never at a raw sqlite3 shell on the live file. See #69603. r)has_live_connectionza connection to z is still open in this process; raw-copying it would cancel that connection's POSIX advisory locks. Close all SessionDB handles first.z&Refusing to raw-copy %s for backup: %sNz %Y%m%d_%H%M%SrrN_z.backup-staging-*z.malformed-backup-*.incomplete*Tr zBReusing existing forensic backup %s (identical to the damaged DB).z"Refusing forensic backup of %s: %sz.backup-staging-z%Could not back up malformed DB %s: %szbackup copy failed: )r0rer7r6rBrGrrrstrftimer!r" itertoolscountrrDglobr8rr'rinforr@r,rrH) r%rBreasonrr8seqpatternoldnewestsrc_idrKs r&_backup_db_filerOs&   [ ) C w '( 2^^F LLA7F S<   '     ! ! # , ,_ =E##w||n4Fug$NOKq!Z!!# ''7<<.8J5'QRSVRW(XY Z2#<<.(9:w||nLkM+C51112sg/I 9AI1I$%I1 AI% I1!/I19I1 II" I1%I.*I11 J":JJ"J"zstate.dbdb_labelc t|dk(st|jdryd}tjt5t t j}ddd|j|jdz|j|jdzf}|jdfgd|g|DD]\}}|r|jr5tj|tjtjzrO|rdnd }d }tjt t"5|duxr|jj%|}|retj&||j)j*t(j,zt(j.z|rt(j0nd zddd|rOtj|tjtjzrt2j5d |||[|jj7drd nd }t9j:|d|rdndd|d|d|d| y#1swYxYw#1swYxYw)u%Refuse-or-repair read-only DB files BEFORE the first connection opens. A stray read-only ``state.db`` / ``-wal`` / ``-shm`` (sudo run, restored backup, copied dotfiles) otherwise surfaces as an opaque "attempt to write a readonly database" inside ``_init_schema``, and the obvious wrong "fix" (deleting the ``-wal``) loses committed transactions. ``chmod u+rw`` repair only inside the Hermes home tree (Hermes owns those files; ``chmod`` fails on files the user doesn't own, bounding the repair exactly); otherwise fail fast naming the file and command. Never deletes/truncates a WAL sidecar — once writable, the normal open checkpoints it. ``:memory:``/``file:`` skipped. Shared with ``kanban_db``. Port of Kilo-Org/kilocode#12508's startup preflight. This preflight: z:memory:zfile:NrrTc3FK|]}|js|dfyw)FN)is_filers r&r'z+preflight_db_writability..,s /hq\]\e\e\gE /hs! !xFrz2%s preflight: repaired read-only %s (chmod u+rw%s)uv Do NOT delete the -wal file — it contains committed data that will be merged into the database once it is writable.z is not writable: directoryfile zf is read-only for this user. Hermes needs read-write access to open the database. Fix with: chmod u+rwz 'z4' (files owned by another user may need sudo/chown).)r>rr0rerrr resolver!r"rDis_dirr4accessR_OKW_OKr8ris_relative_tochmodrst_modeS_IRUSRS_IWUSRS_IXUSRrGrHr#r~OperationalError) r%rQhomer:rr[rUin_scopewal_notes r&preflight_db_writabilityris, 7|z!S\%<% strategies -> promotion window so no writer can commit a change promotion would overwrite. Existing readers make acquisition fail rather than being disturbed (fail closed). Timeout 0: the cross-process lock already serializes repairers, and a partial repair is less safe than "stop the gateway and retry".zBEGIN EXCLUSIVEN)rr~Errorr8r)r%guardrKs r&_exclusive_repair_db_guardrsb):;Tk  MM7 #Ci s;A #A AA AAA  A AA)source_connectiondestination_connectionc t| tj zd fd }tj5}|xs|j t |}|!|jrtjd|xs|j t |}|j|d|tdddy#1swYyxYw)a.Copy one complete SQLite snapshot without replacing either file inode: the online backup API folds committed WAL frames into the source snapshot and writes the destination in one transaction (rolled back if interrupted), so ``state.db`` is never swapped out from under handles that refer to it.cTtjk\rtdddy)Nz/timed out copying SQLite repair snapshot after z.0fs)rirj TimeoutError)_status _remaining_totalrodeadline_secondss r&_check_deadlinez0_copy_database_snapshot.._check_deadlines2 >> x '!PQabePffghi i (r+Nz:SQLite repair backup destination has an active transaction)pagesprogressrl)rr}rr}rr}rNone) rrirjr0 ExitStack enter_contextrrin_transactionr~ProgrammingErrorbackuprm) rdestination_pathrrrownedsource destinationrors @@r&_copy_database_snapshotrs8 D~~"22Hj    i5"Te&9&9,{:S&T ! -2H2W2W**+gh h,c0C0CLQaDb0c  kNg h iiis A9CC c,tfddDS)zW``no such table/column``: FTS5 / core tables not created yet (brand new file mid-init).c3TK|]}|tjv!ywrrrs r&r'z$_schema_not_built..s!RqCHNN$$Rr)z no such tablezno such column)rrKs`r&_schema_not_builtrs R.QR RRr+zUmessages_fts(_trigram|_cjk)?(_data|_idx|_content|_docsize|_config|_segdir|_segments)?z\bTree (\d+)\bzmissing from index (\S+)c|Dcic]\}}}|s t||}}}}|D]}t|}|jdrytj |}|rC|j t|j dd}|rtj|sytj |} | stj| j dryycc}}}w)aTrue when any damaged object named by ``PRAGMA integrity_check`` output lies outside the FTS shadow set: a ``Tree N`` id mapped through ``sqlite_master.rootpage``, an index in ``row N missing from index X``, or the file's own freelist. Unparseable lines are not counted (the FTS wording stays, which is incomplete rather than wrong). #88587: an FTS rebuild cannot repair a canonical b-tree, and the ``.malformed-backup`` it leaves behind is a snapshot of the same damage.FreelistTrNrVF) r}r>r_INTEGRITY_TREE_REsearchrgroup_FTS_OBJECT_RE fullmatch_INTEGRITY_MISSING_INDEX_RE) integrity_lines master_rowsrp_typer"name_by_rootpagelinetexttreemissings r&integrity_damage_is_structuralrs>IOO/"eTBB OO 4y ??: &!((. #''DJJqM(:B?DN44T:-44T: >33GMM!4DE  Ps C#C#c tjt|dd} |j dj Dcgc] }t |}}|j dj Dcgc]}t|d}} |jttjjd|D|S#tj$rYywxYwcc}wcc}w#tj$rY|jytj$rY|jywxYw#|jwxYw) aRead-only ``integrity_check`` + ``sqlite_master`` rootpage map on a fresh connection; ``integrity_damage_is_structural`` over the result. A check that RAISES instead of reporting (a torn page under the walk) is structural too: no FTS-only fixture does that while ``messages``/``sessions`` read cleanly, and the FTS rebuild ladder cannot help. Cannot-open / locked stays False so the caller keeps the FTS path.Tg?)urirnFzASELECT rootpage, type, name FROM sqlite_master WHERE rootpage > 0PRAGMA integrity_checkrc3<K|]}|jywr) splitlines)r#rs r&r'z1state_db_has_structural_damage..s%JDdoo&7%Js)r~connectr rr{fetchalltupler>rertrrrEchain from_iterable)r%rrrliness r&state_db_has_structural_damagers' /8dCP )- O*QQYQYQ[]AuQx] ]$(LL1I$J$S$S$UVqQqTVV )%%%JE%JJK YY ==]V  # #     s^!C!C5C+#C5:C0C5C('C(+ C55D?ED?,E>D??EEc ddlm}m}t|} t j |5|||j dj|j dj}|Dcgc]3}|st|djdk7s&t|d5}}|rdj|ddcdddS|j d jtD])} |j d |d |d j+dt'j(} |j d|j d| dt'j&f|j d| ddt'j&ftD]} |j d|d|d t j*tj,5|j dddd dddycc}w#tj$r]}|j|xs t!|} t#|tj$r| sd |d|cYd}~ccdddSYd}~d}~wwxYw#tj$$r(}|j|s t!|sYd}~d}~wwxYw#tj$r}t!|sdt|jvrWYd}~t j*tj,5|j ddddn #1swYnxYwdddyd|cYd}~t j*tj,5|j ddddn #1swYnxYwcdddSd}~wwxYw#1swYxYw#t j*tj,5|j ddddw#1swYwxYwxYw#1swYyxYw#tj$r}t|cYd}~Sd}~wwxYw)uProbe a DB on a fresh connection. Returns None if healthy, else a reason. Runs the first statement that trips the malformed-schema parse (``PRAGMA journal_mode``), ``integrity_check``, a ``sessions`` read, FTS5 MATCH probes and a rolled-back ``messages`` write — so FTS5 index corruption (reads and ``integrity_check`` pass, every ``INSERT INTO messages`` fails through the FTS triggers) is reported as unhealthy. See #50502. r) SessionDBload_fts5_cjk_extensionzPRAGMA journal_moderr; NrzSELECT COUNT(*) FROM sessionszSELECT 1 FROM z WHERE z MATCH '""' LIMIT 1zfts5 read probe failed on : _hermes_fts_health_probe_zBEGIN IMMEDIATEz>INSERT INTO sessions (id, source, started_at) VALUES (?, ?, ?) _health_probezOINSERT INTO messages (session_id, role, content, timestamp) VALUES (?, ?, ?, ?)user_fts_health_probe INSERT INTO (z) VALUES('flush')z no such tokenizer: cjk_unicode61rzfts5 write probe failed: )rrrrpr0rdr{r|rr>rjoin _FTS_TABLESr~r_is_fts5_unavailable_errorrrreritime_nsrer) r%rrrrowsrproblems fts_tablerKbenignprobe_session_ids r&_db_opens_cleanlyrs@ #7 +D<    %9  $D ) LL. / 8 8 :<< 89BBDD+/Sa1QqT9Jd9RAaD SHSyy"1.9 9  LL8 9 B B D) O  OLL>)GI;Nc!dennp O&";4<<>:JK  - ./ ].MO n.8KTYY[Y[!,"I" |I;a {J[%\]"  ((7-LL,-s9 9  T,, O'AA#F`J[\_J`F&sG,D,DE&"``. An *exc* proving deterministic corruption consumes the persistent repair budget (private ``_repair_attempted`` marker, popped by the caller).T_repair_attemptedrzstate.db repair z: %s)rrGr)reportverbrrKs r& _repair_skiprNsB ;C@&*"#F7O LL#D6.6 Mr+T)rc ddddd}tddt|}|js |d|d<|St|rt |d t |St |5}|s't| d \|d <|d <nd |d<|cdddS|}t|rt |d t |nIt|rt |d dn0t|}t||||}|jd r||d<|jdds|jd r%t|t|jd ddd|S#1swYSxYw)aRepair a state.db whose ``sqlite_master`` is malformed or whose FTS indexes reject writes. Two corruption classes: malformed schema / "duplicate object definition" (even ``PRAGMA`` fails), and FTS write-corruption (reads and ``integrity_check`` pass, writes fail through ``messages_fts*`` triggers). ``_REPAIR_STRATEGIES`` run least-destructive first on a complete snapshot; a success is copied back transactionally, so canonical rows are never modified by a failed attempt. A raw backup is taken first unless ``backup=False``. Serialised across processes (gateway, Desktop backend and CLI open the same file; concurrent ``writable_schema`` surgery is itself a corruption source). Returns ``{repaired, strategy, backup_path, error}``. See #50502. FN)rstrategyr8rg @state_db_repair)phasez does not existrskipped)Trepaired_by_other_processrrzcould not obtain the state.db repair lock (held by another process, or the lock file was unopenable); skipped schema surgery to avoid racing a concurrent repairerza live writer still holds state.db; skipped schema surgery to avoid tearing b-tree pages under a concurrent writer. Stop the gateway (hermes gateway stop) and retry.)rrjournal_mode_beforerr)r)r rrrrr rvrr_probe_journal_mode_for_repair_repair_state_db_schema_lockedrpoprbool)r%rr holding_lockresult before_modes r&repair_state_db_schemarYs+0TRVaefFE):;7mG >> $I_5w -W5FI/QRY/Z[[ #G ,#S !)19Z6z"F:$6$xw#S#S 1 9 ,Nw,W X#7 + -u v9AK3GFSYHSUFzz*%0;,- ::)5 1VZZ 5K "7T&**Z:P5Q RG#SH MI#SH Ms& EB5EEcddlm} t|5}||cdddS#1swYyxYw#tjt f$rYywxYw)uBest-effort journal-mode probe: ``wal``/``delete``, or ``None`` when the file cannot be opened or probed (malformed header, concurrent opener's locks — both expected on the repair path); callers then fall back to ``database.journal_mode``.r)_on_disk_journal_modeN)rvrrrr~rr8)r%rrs r&rrsM7 ' " /d(. / / / MM7 #s# 1% 1.11A  A rcdddlm} |)t|5}|||j}dddn|||j}|r|k7rtj d|||yyy#1swY+xYw#t jtf$r!}tj d||Yd}~yd}~wwxYw)uRe-apply the journal mode after schema surgery. ``conn`` must be the exclusive repair guard connection when called from the repair path: opening a fresh connection AFTER the guard released let a writer still holding the unlinked old ``-wal`` inode coexist with a brand-new ``state.db-wal`` — two generations of one store. The reopen is the hazard, not the mode. A rebuilt file comes back in the default (delete) mode; without this a corruption event silently moves a WAL store out of WAL (the open-time WAL-reset gate never sees a flip made inside repair). Routed through :func:`apply_wal_with_fallback`, not a direct pragma, so it inherits the WAL-reset gate (a vulnerable runtime deliberately keeps DELETE; the journal_mode-changed WARNING is expected there), the macOS-NFS silent-refusal handling and the WAL companions. ``before_mode`` is only for the log comparison; the target is ``database.journal_mode``. Best-effort: the repair already succeeded, so failures log at WARNING. See #89674. See #89393. The transactional promotion already leaves the destination in its pre-repair mode, so on that path this is mostly the WAL-companion re-assertion; the reopen is the hazard, not the mode. See #101064. r)apply_wal_with_fallbackNrPzstate.db repair changed journal_mode %r -> %r (pre-surgery probe %r; restore resolved through apply_wal_with_fallback per database.journal_mode and the WAL-reset gate)zvstate.db repair at %s: post-surgery journal-mode restore failed (%s); verify with PRAGMA journal_mode on the next open) rvrrrr"rGrHr~rr8)r%rrrrafterrKs r&"_restore_journal_mode_after_repairrs&9 F <g& N%/ M N N,D7<<HE 5K/ NNo&{ <0;  N N MM7 #F78? F FFs- A5A)rrrr8r~rr_run_repair_strategiesrrGrHrrrr) r%rrrscratch cleanup_errorbpath backup_error live_guard guard_error space_errorrKs r&rrs 7<<. @AG*733 @FI1suBtC0DE E!)1H.zF:.  -g6|.3E } = 4MMYN4\] ] $G ,+l0I[   4x#.-416AC+l+l 7w? ?K L ;? +l+l F $GW S l+/F& ' "7F 3zz*% f+GWU_`NN#`#)::j#97D6w@SZde::j) n$f]&;="37!;; H[]jkW+l+l 5 F g & 5jkrjssuvyuz3{BEF F+l+l F  =NMM544TUX4Y"[(abeaf&gF7OLL!GMM N$"37!;; H[]jkIU+l+lsJ 4J F%'"I  G9A I 8#J %G6G1"G6#J 1G66J 9I;II II  &JJ  Jcfddlm}g}|gt|D]&}tdD]} |j d&(d j|xsdS#t $rQ}t |tr!|r|dkrtjdYd}~j|j|d |Yd}~od}~wwxYw) zCRemove *path* and every SQLite sidecar; return any cleanup failure.rrX Tr  g?Nrr) rrYr*rangerr8rPermissionErrorrirlr6r)rrYfailuresr+attemptrKs r&rrs(H*)D/* Ry G 4  .    99X  &$& 4c?3 RS JJt$6("SE 233  4sA B0,B+B++B0c|jd|r t||jd|jy)zRun *edit* under ``writable_schema=ON``; bump the schema cookie when it reports a change so live peers discard their cached schema.zPRAGMA writable_schema=ONzPRAGMA writable_schema=OFFN)r{rcommit)redits r&_edit_sqlite_masterr0s4 LL,- vD!LL-.KKMr+c ddlm}||tD]F}tjt j 5|jd|d|ddddHy#1swYSxYw)zFTS5 'rebuild' rewrites each index from the content table: the least- destructive fix for an index that rejects writes while reads work.r)rrrz) VALUES('rebuild')N)rrrr0rer~rer{)rr table_names r&_strategy_rebuild_ftsr:so5D!!V  !9!9 : V LL< |1ZL@ST U V VV V Vs AA( c\t||jd|jy)zintegrity_check reports "wrong # of entries in index" when a B-tree index drifts from its base table; REINDEX rewrites it from canonical rows.REINDEXN)ror{rrs r&_strategy_reindexrEs!!&LLKKMr+c*dfd }t|y)zJDe-duplicate sqlite_master (lowest rowid per type/name wins), keeping FTS.cjdj}|D]\}}}}jd|||ft|S)NzhSELECT type, name, COUNT(*) AS c, MIN(rowid) AS keep FROM sqlite_master GROUP BY type, name HAVING c > 1zFDELETE FROM sqlite_master WHERE type IS ? AND name IS ? AND rowid <> ?)r{rr)dupestype_r"_countr)rs r&_dedupz&_strategy_dedup_schema.._dedupPs] v (* */ . %E4 LLat, . .E{r+Nrr)r)rr"s` r&_strategy_dedup_schemar$Nsf%r+c\tfdtjdy)zDrop all FTS schema and VACUUM; indexes rebuild on the next open. The destructive one, and why strategies run on a scratch copy: on a damaged schema b-tree VACUUM silently drops every table hanging off the unreadable part.c.jdxsdS)Nz9DELETE FROM sqlite_master WHERE name LIKE 'messages_fts%'T)r{rsr&r4z+_strategy_drop_fts_vacuum..`sdll3n&o&wswr+VACUUMN)rror{rs`r&_strategy_drop_fts_vacuumr(\s$wx &LLr+ rebuild_ftszrGrH) r%rr"body success_msg failure_msgrrIrKs r&rrus1C%,dK 1g& $T  &w/F >594 2F: z 2 NN; 0M  $F7O% M  $$ 1XF&{C0 1s- A6 A*A6*A3 /A66B6 #B11B6)r%r)r%rr5r>r Optional[str])r%rrr)r%rr$r>r5r>rIr>)rUr>rr)rr)rarrr)rJrrnfloat)r%rrz Optional[Any])rsqlite3.Connectionrr)rrrr)rr}rr})r%rrr>rz%'Tuple[Optional[str], int, int, int]')r%rrr2)rrrr3)rKrrr)r%rrr)r%rr'Optional[str]')r%rrz'Dict[str, Any]')r%rrr>)r%rrrrr5rr)r%rrz 'List[Path]')r%rr)r}rr)r%rr7rr8rrr)r%rrz&'Tuple[Optional[Path], Optional[str]]')r%rrQr>rr)r%rrnr3rr4)r%rrnr3)rr4rr)r%rrr>rr4) rrrrrOptional[sqlite3.Connection]rr6rrr#r) rDict[str, Any]rr>rr>rKzOptional[BaseException]rr7)r%rrrrr7)r%rrr2rr) r%rrrrr7rr2rr7)rrrr2)r%rrr7rr7)n__doc__ __future__rr0rrrErloggingr4rerr~rripathlibrtypingrrrrr rr hermes_startup_watchdogr rr hermes_state_commonr rrrr getLoggerrGrmrr_MAX_MALFORMED_BACKUPSr(rrrrrrr*r:r@rLrVr_rgrpcontextmanagerrvry_release_auto_maintenance_lockrrrrrrrrrrrrrr rr'r,r@rOrirprrrorrrrrrcompilerrrrrrrrrrrrrrrrr$r(r.rr/r+r&rEs0 #   33,;1   > *3C0#$4! ';#!2#J ^Y $C D:? 82" ((V "6 I_ N ] >n nE:*>4]$ _]aK6=9OV.82v@J$5N@C*58Q    0OSSWi/Ki4Pi\`i.S\ RZZ 12(bjj)DE.Y0L^h=ACL [_ FHbfKl Kl"Kl,:KlQ^KlKl\'(V &)+i46')Z')+Y,.2kmqs r+