Files and schema
The versioned operations.db lives beside state.json under /var/lib/local-server-observability. SQLite WAL and a busy timeout support bounded readers and minute writes. Database permissions are 0600. Version-one tables are created additively for metadata, snapshots, samples, daily summaries, events, incidents, transitions, and account preferences. Existing issue IDs and reports remain in state.json.
Scheduled verification
The daily workflow uses SQLite backup to obtain a consistent operations.db snapshot, encrypts it, decrypts to a temporary file, compares bytes, and runs PRAGMA integrity_check. Encrypted detailed artifacts have 90-day retention. Daily summaries and incident transitions are exported to the encrypted indefinite issue-history archive. Plaintext temporary files are cleaned.
Rollout and restoration
The preflight collector validates incoming configuration and writes initial scores using the retained database before presentation activation. After activation the pipeline imports available history and validates every expected system. To recover operational state, stop the portal collector, restore an authenticated encrypted backup into a staging directory, verify integrity and schema, preserve the current DB/WAL/SHM files, replace the database with correct ownership, and restart. Use the full existing workspace restoration runbook for Grafana and issue state.
Production ingestion log coverage
The root-owned Observe cloud log collector queries the three active NutsNews Cloudflare ingestion shards every five minutes. All three must have authentic activity within one hour, and the collector observation must be newer than twenty minutes. Missing streams, authentication errors and stale results remain unverified. Only a scope and timestamps are published; log contents and credentials remain private. Coverage never implies successful ingestion. The backend metrics timer refreshes every minute to satisfy unchanged freshness thresholds. No paid services are enabled. Stop observe-cloud-logs.timer to disable this evidence source; existing application processes are unaffected.
Inactive worker recovery
Scheduler and translation recovery copy deployed files and configuration without starting or restarting live containers. Sequential disposable stacks restore the actual database and broker topology. Scheduler verification requires existing restored leases, new isolated leases and messages published from restored feed data. Translation verifies restored completed-message replay and a real isolated model inference. Only successful current-revision drills earn recovery evidence; live readiness and production throughput checks remain independent. Missing components or failed startup remain failures. The shared two-CPU/eight-GiB limit and network isolation apply.
The disposable dependency mapper supports both deployed backend URL settings (BACKEND_API_URL and BACKEND_API_BASE_URL), database, broker and model endpoints. It changes only the restored environment and preserves live configuration. A missing mapping cannot escape the private network; the drill fails instead. Selected recovery runs still refresh deployment identities for every registered application between drills, so testing one application does not leave the others unobserved. Observe also compares its own running release directly with the tested revision, invalidating older restore proof immediately after activation rather than waiting for the next poll. Translation replay uses its own stored message, entity and payload-reference identity contract; persistence retains its canonical payload fingerprint check. Successful model startup alone does not satisfy replay verification. A busy shared recovery lock is recorded as a skipped run using exit status 75, while real runner errors remain failures.