README.md (11413B)
1 # radroots_storage_sqlite 2 3 SQLite storage backend for Radroots. 4 5 Startup and migration preserve typed `SpaceInsufficient` for numeric SQLite 6 capacity failures and typed capacity I/O failures. Existing corruption and 7 validation errors remain distinct. Capacity does not prove rollback: inspect 8 the original store before retrying, and retain pending or ambiguous work. 9 10 Projection document inventory uses the existing canonical projection table. 11 Each page releases its read snapshot before returning a continuation. Metadata 12 is bounded before decoding; the 16 MiB payload budget is charged before each 13 value is loaded, including values that fail digest verification. Corrupt 14 payloads retain an explicit locator, while unpageable corrupt keys or 15 generations fail the query. Cross-page mutation fencing remains caller-owned. 16 17 Runtime schema v15 permits late verified authored signatures to remain durable 18 alongside a signing stop. The logical snapshot preserves cancelled or terminal 19 state; a separate SQL stop column preserves the original signed/raw constraint. 20 Prior snapshots, receipt identities, migration bytes and checksums remain 21 unchanged. Older schema policies reject the new version. An update guard prevents 22 replacement of the first signed bytes or removal of the retained signing stop. 23 24 Recording a late fact verifies the backend's immutable original claim receipt 25 inside the same transaction as the artifact, eligible delivery bindings and new 26 receipt. Expiry does not invalidate evidence, and evidence does not renew work 27 authority. Explicitly stopped delivery plans remain stopped. A success receipt 28 is returned only after the actual SQLite COMMIT. 29 30 The backend owns separate `runtime.sqlite` and `private.sqlite` files. Writable 31 opens hold a process advisory lock, use WAL with bounded busy handling, and 32 apply only the governed forward migrations. Fresh stores require a 33 host-supplied `SourceGeneration` and creation timestamp; the crate never reads 34 hidden entropy or a wall clock. 35 36 Runtime schema v14 adds generic schema/scope metadata and an index for bounded 37 draft-head queries. Migration preserves every original revision snapshot, 38 including corrupt historical evidence. Known foreign schemas and scopes are 39 filtered before decoding; unknown historical schemas expose only an opaque 40 author-bound corruption locator. Each page releases its read snapshot before 41 returning a continuation and reads at most 16 MiB of serialized snapshots. 42 43 Draft submission uses the existing WAL/FULL writer and authored tables. One 44 transaction writes the immutable intent, operation, artifacts, delivery plans, 45 ordinary Prepare receipt and source-association receipt. Exact replay precedes 46 source-head CAS, including after reopen or later editing. Existing atomic 47 receipt snapshots retain their 4 MiB limit; oversized submissions fail without 48 partial writes. No additional public database or connection API is introduced. 49 50 Every connection in both owned pools uses `synchronous=FULL` and requests 51 `fullfsync=ON`; writable stores retain WAL. Authored append acknowledges only 52 after a successful SQLite COMMIT, or an exact replay of an already committed 53 revision. Connection-policy validation rejects a missing full-sync request. 54 The existing owner supplies this policy without a second database or journal. 55 56 Qualification covers actual deferred-constraint COMMIT failure, bounded SQLite 57 page-capacity exhaustion, lock contention, denied writes, closed/read-only 58 stores, and child process termination before and after acknowledgment. Failed writes retain the 59 previous revision and return a typed failure without a success receipt. These 60 tests do not fill the host disk or establish physical power-loss survival. 61 SQLite requests `F_FULLFSYNC` where its VFS supports it and may fall back to 62 `fsync`; actual filesystem, device and power-cut behavior remains a separate 63 host qualification. Native protected-data policy remains the host's concern. 64 65 Backend status and the last integrity result are passive. Hosts invoke 66 `check_integrity` explicitly with their own positive timestamp when they want 67 full SQLite and foreign-key validation across both owned files. `close` drains 68 both pools and releases writable authority explicitly and idempotently. 69 70 Backup capture requires an explicit existing host-owned root configured with 71 `OpenOptions::with_backup_root`. V1 capture creates a deterministic staging 72 bundle, uses SQLite's online snapshot mechanism for each policy-selected 73 member, synchronizes captured files and directories, and returns exact lengths 74 and SHA-256 digests in the backend-neutral manifest. Protected storage is 75 included only when the plan requests it explicitly. 76 77 Restore first copies a finalized bundle into create-new files adjacent to the 78 live databases and verifies their exact digests, schema catalogs, SQLite 79 integrity, and foreign keys without changing live state. Finalization closes 80 the owned pools while retaining writer authority, persists a versioned recovery 81 marker, atomically replaces each policy-selected member, and requires callers 82 to reopen the closed backend. Writable open completes an interrupted marked 83 replacement before opening connections; read-only open fails closed until that 84 recovery is complete. 85 86 Legacy migration starts only from an explicit, forward-only import plan on an 87 open writable backend. Before any target mutation, the backend captures every 88 caller-identified predecessor database through SQLite's WAL-consistent online 89 snapshot operation into a create-new staging directory, verifies SQLite and 90 foreign-key integrity, records exact source provenance, lengths, and SHA-256 91 digests in a mode-`0600` manifest, synchronizes the evidence, and atomically 92 finalizes the immutable bundle. Import identities and timestamps are supplied 93 by the host; collisions, owned-database aliases, and unsupported paths fail 94 closed. 95 96 Classification revalidates the finalized manifest, member inventory, hashes, 97 SQLite integrity, and foreign keys before inspecting any schema. Event-store 98 versions 1 through 4 require the exact governed catalog and, when present, the 99 exact contiguous migration ledger; outbox, private, and Studio predecessors 100 require their exact catalog and application version. Unknown objects, mixed 101 source families, checksum drift, and newer schemas fail closed. Studio records 102 are explicitly classified for host handoff and are never imported into SDK 103 storage. 104 105 Beginning an import writes only governed recovery metadata. Runtime schema v6 106 binds one retained import journal to the target generation, finalized manifest, 107 and ordered classification digest, plus one exact member row per predecessor 108 source. SQLite guards immutable identity, one-shot conflicts, legal monotonic 109 state transitions, timestamps, staged counts, and retained audit history. 110 Repeated begin calls resume the exact journal; conflicting attempts fail 111 closed. 112 113 Runtime schema v7 adds isolated, append-only event-import staging. Each bounded 114 page revalidates the immutable evidence, decodes and identifier-checks the 115 legacy signed events, preserves their exact JSON and untrusted predecessor 116 admission evidence, and atomically advances an eight-byte legacy sequence 117 cursor with its durable row count. Restart resumes after that exact cursor and 118 a completed retry is a no-op. Staging never mutates the live canonical event 119 table or upgrades predecessor verification claims. 120 121 Runtime schema v8 stages the predecessor outbox as an ordered five-table 122 graph. A table-discriminated cursor advances through operations, events, 123 delivery plans, targets, and attempts; every child is accepted only after its 124 exact parent, and attempts must bind a target from the same plan. Governed 125 column-order JSON-array records preserve nullable and scalar predecessor data 126 without writing the live operation journal, outbox, or delivery evidence. 127 128 Private schema v2 stages secret-bearing predecessor records only inside 129 `private.sqlite`. Runtime and private commits use an explicit replay protocol: 130 enter runtime staging, idempotently commit and byte-verify one private page, 131 then compare-and-swap the runtime table cursor and count. A process loss after 132 the private commit therefore replays the exact page from the old runtime cursor 133 without duplicating counts or exposing secret-bearing staging in 134 `runtime.sqlite`. 135 136 Classified Studio state is never imported into either owned database. The 137 backend instead returns an immutable evidence descriptor bound to the import, 138 target generation, manifest, source digest, schema catalog, and byte length. 139 Only an exact host receipt carrying that handoff identity and a non-zero opaque 140 host-store commitment advances the Studio journal member to ready; exact retry 141 is idempotent and conflicting acknowledgement fails closed. 142 143 Before finalization, `validate_legacy_import` revalidates the immutable evidence, 144 requires every member to be ready, matches each staged count to its exact source 145 count, and hashes framed member cursors plus every runtime/private staging row 146 under write-stable snapshots. The returned digest is the deterministic commit 147 identity; validation itself mutates nothing. 148 149 Finalization seals that identity private-first, then completes the runtime 150 journal and all members in one transaction. A crash after the private marker 151 replays and verifies it while the runtime journal remains ready; a lost success 152 response reconstructs the exact receipt from both commit markers. Immutable 153 legacy staging remains retained as owned migration evidence, while live product 154 tables are not dual-written and predecessor evidence is not deleted. 155 156 The qualification matrix includes every supported predecessor family, mixed 157 four-source operation, bounded page interruption and reopen, invalid-row 158 rollback, unsupported-schema and identity conflicts, private-first recovery, 159 lost-success retry, source retention, and live-table isolation. 160 161 ```rust,no_run 162 use radroots_storage::event::SourceGeneration; 163 use radroots_storage_sqlite::{OpenMode, OpenOptions, Paths, SqliteStorage}; 164 165 # async fn open(directory: &std::path::Path) -> Result<(), radroots_storage_sqlite::Error> { 166 let paths = Paths::from_directory(directory)?; 167 let generation = SourceGeneration::new([7; 32]).expect("non-zero generation"); 168 let storage = SqliteStorage::open( 169 OpenOptions::new(paths, OpenMode::Create) 170 .with_source_generation(generation, 1_700_000_000_000)?, 171 ).await?; 172 # drop(storage); 173 # Ok(()) 174 # } 175 ``` 176 177 Atomic paired authored revisions use the existing runtime revision tables in one 178 `BEGIN IMMEDIATE` transaction. Both insertions commit together; mismatched or 179 partial replay rolls back. Complete replay returns immutable historical rows and 180 does not establish current application authority. No migration or second storage 181 owner is introduced. Callers may recover an unknown commit result by replaying 182 the exact pair. 183 184 Open inspects both existing database schemas through read-only connections before 185 configuring writable connections or applying either pending migration. A future 186 version, foreign namespace, invalid catalog, ineligible authored migration or 187 incompatible source generation therefore leaves both original files intact. Each accepted migration still 188 rechecks metadata and applies its pending suffix atomically in its own database; 189 this does not claim a cross-database atomic commit. An interrupted compatible 190 upgrade resumes from its committed version on the next open.