Scheduler and state
The infrastructure repository owns .github/workflows/inspect.yml. Runs share the host deployment concurrency group. The inspection cursor starts at the previous successful execution, capped at seven days; a cap creates incomplete coverage. State is atomically written to /var/lib/local-server-observability/state.json. Report summaries and issue transitions are retained indefinitely; detailed diagnostic evidence is pruned after 90 days.
Deterministic checks
Resource checks use versioned thresholds: CPU and memory 90%, root filesystem 85%, public endpoint availability 99%, and TLS seven days. Local service health, restart counts and backup status use existing exporters. Explicit metric inspectionThreshold values apply to every host. Fallback limits require matching metric units and exact utilization, availability or certificate names: available-memory bytes and disk-throughput bytes per second are informational samples, not percentage violations. Genuine memory/disk utilization and Cloud alerts remain evaluated separately. Missing instrumentation remains an explicit incomplete check. Log inspection uses fixed selectors and a ten-query, 500-entry-per-query budget. Full forward pages defer all entries at their final timestamp and re-read that boundary, so tied logs cannot be lost or double-counted. A short final page proves exhaustion; a saturated single timestamp, exhausted budget, malformed response or failed query remains incomplete. A separate successful query must also prove fresh source activity. Evidence contains approved classifications and counts, never raw lines. Recurring-error thresholds and score weights are unchanged.
AI and backups
After deterministic checks, an idle-only local Qwen request may summarize up to twelve finding groups. It runs once with a 60-second deadline, within the five-minute total allowance. Active inference connections, failure or invalid evidence references produce a template fallback. AI never changes lifecycle state. AES-256-GCM encrypted evidence backups are stored as 90-day Actions artifacts; encrypted summaries and transition histories are retained in the issue-history GitHub release. Each backup is decrypted and compared before upload.
Verified backup restoration
The daily pipeline takes a consistent SQLite snapshot of owner Grafana, encrypts it separately, decrypts it in the runner and requires PRAGMA integrity_check to return ok. Both report bytes and database bytes must match their originals. Detailed evidence and owner database snapshots use 90-day encrypted artifacts. Indefinite summary archives live in a draft GitHub release so backup publication adds no release notifications. To recover, download the chosen Actions artifact using an authorized GitHub session, supply REPORT_BACKUP_KEY privately, and run services/workspace/backup.mjs decrypt input.enc output. Validate state JSON and SQLite integrity before replacing files. Stop only local-server-observability and observe-grafana-owner for replacement, preserve a rollback copy, restore ownership to observability, restart those two services and repeat the Qwen compatibility checks. Never replace the live SQLite database while Grafana is writing it.