Baseline jobs
Adopt existing database and home-server image backup timers, health checks, and backup-status metric generation. Inspect the installed schedules and destinations before managing them.
Application rollback
The deployment stores versioned portal releases and the previous active target. Rollback reactivates a retained verified release and checks both the portal and Qwen connection. It must not overwrite application data.
Configuration recovery
Sensitive configuration snapshots are stored outside public Git. Restore only the affected files, validate them, and reload the relevant service. Preserve unrelated tunnel routes and the production AI environment.
Full-host recovery
Restore or install Ubuntu, establish network and SSH access, enroll the tunnel connector, then run the provisioning workflow. Model downloads and data restoration may require separate elapsed time.
Local backup panels
A dedicated dashboard uses existing home_server_backup and nutsnews_db_backup metrics for outcomes, attempt age, archive size, duration, cloud copies, and status freshness. Local alert rules surface failed backup outcomes and stale status collection without changing any backup jobs.
Recovery controller
The root-owned observe-recovery.timer polls deployments every fifteen minutes. The service shares /run/observe-recovery.lock, CPUQuota=200%, MemoryMax=8G, MemorySwapMax=0 and an isolated loopback network. Successful unchanged applications run weekly; failed attempts back off six hours. An application deployment invalidates its prior evidence. Only the controller writes /var/lib/observe-recovery/public/evidence.json; Observe reads sanitized status, scope and timestamps.
Backup and restore path
Capture the deployed revision and its real saved data; encrypt with AES-256-GCM, upload to the existing homebackup encrypted remote, download and validate the encrypted hash and authentication tag, and restore into a disposable filesystem. Static sites rebuild archived deployed source and deployment configuration. KubeQuest reads restored SQLite progress through its private application route. Qwen instances execute isolated CPU inference using restored model blobs. Observe boots restored operations history, owner Grafana database, saved dashboard files and provisioning configuration. Restored provisioning paths point only inside the disposable filesystem; dashboard counts and individual dashboard reads must pass through the running Grafana API. Backend and Fantasy use restored PostgreSQL data and real application readiness and read routes. Readiness and full application behavior receive separate evidence.
Cloud database backup
The adopted Supabase backup now archives the complete public schema and data, authentication schema definitions without rows, roles without passwords, and required extension declarations. The existing encrypted destination, schedule and retention remain unchanged. Use existing credentials and the production session pooler; never reset a password to repair a backup. The repeatable verify-cloud-database.mjs drill restores role names without production privileges or memberships, then the authentication schema, and injects extension prerequisites immediately after the public schema is recreated. Run it under the shared recovery lock with CPUQuota=200%, MemoryMax=8G and MemorySwapMax=0. Validate checksums, relational integrity and saved data.
Credentials and operator access
Recovery credentials and the encryption key live under root-only /etc/observe-recovery. Key versions are preserved under their hash in the existing encrypted cloud destination. Restore the matching key privately; never place a key, database dump or detailed diagnostic evidence in a public repository. The backend exporter uses a pinned SSH host key and a forced-command identity with fixed read-only exports. It cannot select arbitrary commands, paths or restore targets.
Verified database repair
On 2026-09-20, the existing production session-pooler credential repaired the authentication failure. The downloaded cloud backup nutsnews-db-2026-09-20_00-41-26 passed checksum validation and restored its complete public schema and 2,449 articles into disposable PostgreSQL. Backend snapshot 9b3b0868 passed its existing integrity verifier; an ExecStartPost verifier removes the former backup-to-verification timing gap. No passwords, backup destinations or retention policies changed.
Dependency preparation boundary
The preparation phase downloads locked npm dependencies with lifecycle scripts disabled. No application or package install hooks run with network access. Builds and restored application behavior run only in the disposable loopback-only network namespace.
Personal website artifact recovery
The ramideltoro.com Netlify manual deployment is captured by immutable published deployment ID, not an assumed Git commit. Archive every file with its provider checksum plus hosting settings; reject deployments with server or edge functions. Download the encrypted cloud backup, verify all restored file hashes and serve every archived file inside the private recovery namespace. No production deployment is changed. NETLIFY_AUTH_TOKEN is stored only in the existing root-only recovery credential file.
Observe log continuity
Each completed operations collection emits a bounded structured event containing only its event name and system count. This proves actual application work independently of dashboard visits; the existing scoped Loki selector and freshness requirement remain unchanged.
Shared cloud backup safety
Image retention and image freshness inventory accept only the current hostname followed by the exact YYYY-MM-DD_HH-MM-SS directory format. Database and Observe recovery folders are never image-retention candidates. The existing policy of retaining the newest verified image backup remains unchanged. Both scripts are versioned and installed through the validated infrastructure release.
Isolated worker recovery
The fixed backend exporter captures deployed worker files/configuration, populated PostgreSQL data and roles without passwords, sanitized broker topology and the broker runtime. All queues, including canary/retry/dead-letter queues, must be empty before and after capture. Restore the downloaded encrypted archive in the existing shared-lock namespace with 2 CPUs, 8 GiB RAM and no swap. Bind restored Alpine musl/Erlang/RabbitMQ libraries read-only inside that namespace; no nested privileged container or production network is required. Preserve database roles and ACLs. Require real worker readiness, then publish a schema-valid synthetic carrier with an actual completed inbox identity. Require duplicate recognition, zero queued replay messages, unchanged inbox count and byte-equivalent completed record. Persistence/publication use the restored backend API; approval restores Qwen and executes isolated inference. This proves durable-state recovery and duplicate replay, not new-work throughput or cutover readiness. Weekly/current-revision/eight-day evidence rules are unchanged.
Live image capture boundaries
Exclude virtual filesystems, the actively written system journal and temporary recovery staging from the root image. Keep ordinary application data, configuration and existing image retention/destination intact. Compression uses two threads at level six to reduce competition with production. An archive read error still fails the job; no general changed-file error suppression is used. Prior cloud images are removed only after a verified replacement upload.
Latest-attempt truth
The backup status collector reads the durable latest-attempt state. A retained older cloud image proves availability of that image but must never turn a newer failed attempt into success.
Persistence also requires the original backed-up upstream outbox envelope and payload, with its aggregate/payload fingerprint matching the restored completed inbox row before replay. A synthetic carrier must not bypass that conflict check. Deployment identities are re-probed between weekly drills so earlier results stay current during a long serialized run.
ShowAlgo Firebase artifact recovery
The observe-recovery-showalgo service account has only roles/firebasehosting.viewer on the showalgo project. The approved JSON credential is stored as SHOWALGO_FIREBASE_SERVICE_ACCOUNT_B64 in the central credentials file and root-only /etc/observe-recovery/credentials.env; never commit it or include it in an archive. OAuth tokens are short-lived and retained only in memory. Read the current finalized live version, its complete paginated file inventory and immutable ServingConfig. Download from the fixed showalgo.web.app origin without credentials, verifying each compressed SHA-256 against the provider inventory. CDN-decompressed binary files are recompressed at Firebase’s level-nine gzip format and must match that same hash. Capture rejects missing files, checksum failures, duplicate paths, unsupported serving features and a concurrent deployment. After encrypted cloud roundtrip, the isolated verifier checks every file, directory index routes, missing-file behavior and the supported cache headers. This proves recovery of the deployed static artifact and hosting configuration; it does not assert the old Git checkout corresponds to that deployment or prove a source rebuild. Unrecognized serving configuration remains failed until explicitly supported and tested. Weekly/revision/expiry and resource limits are unchanged.
Raspberry off-host snapshots
The existing serialized recovery runner captures the root filesystem plus the separate boot files through a restricted SSH key and a fixed read-only exporter. Volatile files, package caches and swap contents are excluded; application data, installed packages and service configuration are retained. The runner encrypts, uploads, downloads and checks the archive and required members before recording readiness. Evidence renews daily, expires after 27 hours, and is invalidated by configuration/revision mismatch. Full restore evidence is not created. The local runner retains its 2 CPU / 8 GiB limits; Pi export uses low-priority streaming with no local snapshot file. Existing retention and application state remain unchanged.
Cloudflare ingestion restoration
The nutsnews-cloud-workers provider archives the deployed modules and configuration of shards 0–2, actual shared KV values, and a consistent backend PostgreSQL export with application configuration and dependencies. Unsupported bindings, split deployments, incomplete KV enumeration and empty sources fail closed. The downloaded encrypted backup runs in a private network namespace with PostgreSQL, the restored Python backend and pinned Miniflare. Each shard must serve saved public-feed articles, recognize restored article duplicates and persist its run without new article identities, paid/local AI calls or external actions. Captured KV expiration metadata is preserved in the archive; the short test retains point-in-time values. Evidence remains independent of production ingestion freshness.
Deployment observation refresh
A nonoverlapping five-minute probe refreshes deployment revisions during long capture and cloud-transfer steps. Serialized atomic writes prevent evidence-file races. Shutdown waits for the probe before the final observation. Failed observations cannot renew recovery proof, and the eight-day evidence expiry and revision checks remain unchanged.