What is saved
Health samples, deployment events, incident history, and owner preferences are saved on the local server. Daily summaries and incident transitions are retained indefinitely; detailed samples and events are kept for 90 days. Existing issues and reports are preserved.
Backups
The daily GitHub inspection makes a consistent encrypted database backup and verifies it can be decrypted and read. Long-term summaries and transitions are encrypted separately. The portal reports this as an integrity test, not a full production restore.
Safe releases
Collectors and storage are validated before the new pages are activated. Rolling back the portal keeps the database and issue history. Production restores and automatic remediation are not part of this release.
Production ingestion log coverage
The root-owned Observe cloud log collector queries the three active NutsNews Cloudflare ingestion shards every five minutes. All three must have authentic activity within one hour, and the collector observation must be newer than twenty minutes. Missing streams, authentication errors and stale results remain unverified. Only a scope and timestamps are published; log contents and credentials remain private. Coverage never implies successful ingestion. The backend metrics timer refreshes every minute to satisfy unchanged freshness thresholds. No paid services are enabled. Stop observe-cloud-logs.timer to disable this evidence source; existing application processes are unaffected.
Inactive worker recovery
Scheduler and translation recovery copy deployed files and configuration without starting or restarting live containers. Sequential disposable stacks restore the actual database and broker topology. Scheduler verification requires existing restored leases, new isolated leases and messages published from restored feed data. Translation verifies restored completed-message replay and a real isolated model inference. Only successful current-revision drills earn recovery evidence; live readiness and production throughput checks remain independent. Missing components or failed startup remain failures. The shared two-CPU/eight-GiB limit and network isolation apply.
The disposable dependency mapper supports both deployed backend URL settings (BACKEND_API_URL and BACKEND_API_BASE_URL), database, broker and model endpoints. It changes only the restored environment and preserves live configuration. A missing mapping cannot escape the private network; the drill fails instead. Selected recovery runs still refresh deployment identities for every registered application between drills, so testing one application does not leave the others unobserved. Observe also compares its own running release directly with the tested revision, invalidating older restore proof immediately after activation rather than waiting for the next poll. Translation replay uses its own stored message, entity and payload-reference identity contract; persistence retains its canonical payload fingerprint check. Successful model startup alone does not satisfy replay verification. A busy shared recovery lock is recorded as a skipped run using exit status 75, while real runner errors remain failures.