Cluster Manager environment advisories (internal)

Engineering reference for the watchdog in core/cluster-mgr/server/src/services/advisories/.

Endpoint

GET /api/environment/advisories (admin Bearer required)
In-memory active cache; no persistence in v1
Must live under /api/environment (ingress prefix for cluster-manager BFF). Unlisted /api/* paths route to core-api.

Routine catalog

code	category	interval	Source
`heat.advisory.security.core-auth-local-exposed`	security	5 min	`system.core.auth_mode` + outbound internet probe
`heat.advisory.updates.platform-image-tag-mismatch`	updates	5 min	`kubectl` image tags: `core-api` reference vs platform Deployment allowlist (runners excluded)
`heat.advisory.performance.runner-output-gzip-disabled`	performance	5 min	`runner_config.gzip_node_outputs` not `true`
`heat.advisory.availability.nats-queue-unavailable`	availability	5 min	`kubectl` pods `app=heat-nats-queue` in `heat` namespace
`heat.advisory.availability.object-store-unavailable`	availability	5 min	`kubectl` pods `app=heat-managed-object-store`
`heat.advisory.availability.postgres-unavailable`	availability	5 min	`kubectl` pods with `postgres` in pod name
`heat.advisory.performance.pod-oom-killed-recent`	performance	5 min	`kubectl` container `lastState`/`state` terminated `OOMKilled` within 15 min
`heat.advisory.availability.node-count-low`	availability	5 min	`kubectl get nodes`; high at 1 node, medium at 2, clear at 3+
`heat.advisory.performance.platform-memory-below-spec`	performance	5 min	Sum of node memory capacity; high ≤32 GiB, medium ≤48 GiB
`heat.advisory.performance.platform-cpu-below-spec`	performance	5 min	Sum of node CPU capacity; high <6 cores, medium <8 cores
`heat.advisory.performance.node-resources-below-spec`	performance	5 min	Per-node memory (<12 GiB high, ≤16 GiB medium) and CPU (<2 high)
`heat.advisory.performance.object-store-pvc-below-spec`	performance	5 min	`heat-managed-object-store` PVC size; high ≤256 GiB, medium ≤512 GiB, low ≤1024 GiB
`heat.advisory.performance.core-api-exceptions-recent`	performance	5 min	`serviceLogCollector` cache: Core API ledger exceptions in `SERVICE_LOG_LOOKBACK_MS`
`heat.advisory.performance.heat-auth-exceptions-recent`	performance	5 min	Same for HEAT Auth `/_logs/auth`
`heat.advisory.performance.v2-api-errors-recent`	performance	5 min	`kubectl logs` error-line matches for `heat-v2-api`
`heat.advisory.performance.runner-type-limit-high:<name>`	performance	5 min	Core API runner type CPU or memory limit >40% of average node capacity (`kubectl get nodes` + `GET /api/RunnerTypes`)

Service log collector

serviceLogCollector.ts polls every 60 s (default):

GET http://core-api/_logs/coreapi
GET http://heat-auth/_logs/auth
kubectl logs -n heat deployment/heat-v2-api --since=5m --tail=500

Exposed via GET /api/environment/service-logs (admin). Advisory routines read getServiceLogSnapshots() from context; they do not call kubectl or ledger HTTP directly.

Operator one-pager: 16-platform-service-logs.md

Adding a routine

Create routines/<name>.ts exporting an AdvisoryRoutine
Register in registry.ts
Use unique heat.advisory.<category>.<slug> code
Add unit test with mocked AdvisoryContext
Update this catalog and 14-environment-advisories.md

Env vars

Variable	Default
`ADVISORY_WATCHDOG_ENABLED`	true
`ADVISORY_WATCHDOG_INTERVAL_MS`	300000
`ADVISORY_INTERNET_PROBE_URL`	https://www.microsoft.com
`ADVISORY_INTERNET_PROBE_CACHE_MS`	600000
`HEAT_NAMESPACE`	heat
`ADVISORY_PLATFORM_DEPLOYMENTS`	Built-in allowlist (comma-separated override)
`ADVISORY_OOM_LOOKBACK_MS`	900000 (15 min)
`SERVICE_LOG_COLLECTOR_ENABLED`	true
`SERVICE_LOG_COLLECTOR_INTERVAL_MS`	60000
`SERVICE_LOG_LOOKBACK_MS`	900000 (15 min)
`SERVICE_LOG_MAX_CACHED_ENTRIES`	500
`SERVICE_LOG_KUBECTL_SINCE`	5m
`SERVICE_LOG_KUBECTL_TAIL`	500
`HEAT_AUTH_URL`	cluster: `http://heat-auth`; dev: `http://localhost:5168`

Platform image tag allowlist (default)

heat-v2-api, heat-auth, cluster-manager, heat-scheduler, heat-auditor, heat-dashboard, heat-legacy-dashboard, heat-legacy-facade, heat-legacy-interposer, heat-external-api-v1, heat-ingest-controller, heat-docs-kb. Reference deployment: core-api. Runners and project-*-worker deployments are not included.

Compliance tags

Static strings on each AdvisoryRoutine.complianceTags. Not validated against a registry in v1.