Documentation ›Operations
OPERATIONS / TECHNICAL

Verified applications and server coverage

GitHub and runtime inventory, Raspberry receiver enrollment, application dashboards, and evidence boundaries.

Inventory verification

The September 19 audit enumerates all repositories accessible under ramideltoro, recent deployment records, Pages configuration, latest workflow outcomes, Cloudflare scripts, five SSH host inventories, container labels and public HTTP endpoints. A workflow success alone is never deployment proof. config/inventory.json stores canonical hosting and repository mappings; config/operations.json supplies health checks and dependency edges. Container revision snapshots are explicitly dated.

Telemetry paths

Local collector port 4311 exports systemd active state, restart count, memory and cumulative CPU seconds for KubeQuest, Fantasy Qwen, Grafana and existing services. Loopback KubeQuest /healthz checks status, application identity, labReady and Google configuration; Fantasy Ollama /api/tags verifies an available model without inference calls. HTTPS probes run every minute independently of browser visits. No response payload or prompt text is retained.

Raspberry uses the existing official Grafana Alloy packaging and the Git-managed ansible/raspberry.yml enrollment play. It writes to the existing Cloud metrics and Loki stores using restricted environment files. Labels are instance=rpi4; journal adds project=raspberry; Alloy assigns job=loki.source.journal.raspberry. Inspection selects instance and project, independent of the generated job label. A minute hardware timer publishes atomic textfiles. Alloy binds loopback and uses CPUQuota=35%, MemoryMax=384M and a low scheduling priority. Enrollment does not alter receiver services or reboot the host.

Worker dashboards

Eight per-stage dashboards expose scrape availability, readiness, expected production ownership, in-flight work, consumers and stage event rates using fixed job and service selectors. Missing readiness metrics remain unknown. Shadow ownership stays expected_active=0. Existing Cloudflare workers and production ingestion remain independent. RabbitMQ dashboards show ready and unacknowledged messages plus acknowledgement rate.

Deployment

Update all three wiki modes and source-lock fingerprints before release. Merge documentation first, then infrastructure and portal changes; deploy the exact portal SHA through the serialized GitHub coordinator. Verify Cloud sample freshness, public health, dashboard datasource queries, five-host coverage, and Google protection for owner routes.

Private investigation

New portal dashboards have native owner copies with the Cloud metrics datasource and host-scoped private logs where applicable. Raspberry and Mookie are accepted explicit scopes in the authenticated log API. Public queries remain registered and anonymous. HTTPS latency warns at 2 seconds and fails at the 8-second probe budget. Host checks and HTTP checks remain separate evidence.

Production versus shadow evidence

Cloudflare production ingestion uses the backend-owned legacy-worker scheduled-run outcomes and success timestamps, scoped to job=nutsnews-backend-host and instance=backend.nutsnews.com. The similarly named nutsnews_worker_last_success_timestamp_seconds belongs to a shadow fetcher and must never be used as production success evidence. Scheduled-success age is displayed without an unverified cadence threshold. A regression test preserves this ownership separation.

ON THIS PAGE

THREE WAYS TO UNDERSTAND

Same system.
Your level of detail.

Switch reading modes above to find the explanation that fits.

Explore Expert →

Keep exploring

Public observability workspace

Public native dashboards, server navigation and owner changes.

↗
Backups & recovery

Protect existing data and know which recovery path to use.

↗