Health API setup

Shift fleet

Shift fleet (Targets, and the same field on Settings → Monitor) picks the default primary probe origin for the watch. It does not migrate applications onto other servers, change DNS, or change who hosts an app. It only answers: which public IP should run the healthy-interval GET for targets that still say “fleet default.”

• Default is denver (this process on addo-primary-den).
• Cloud Linux agents with an ingest token appear in the list. Offline picks fall back to Denver until that agent checks in.
• When primary is a cloud box, Denver is not the scheduled primary. It may still be the extra origin so two IPs can confirm down.
• Impact tiles on Targets update after a shift: they show the new primary plus assigned remotes and planned GET/hour.

The control is for operators who need the watcher off Denver (maintenance, or a second region) without rewriting every target. Leave per-target pins on apps that must stay on a specific origin.

How probes run

The monitor is the scheduler. Agents do not independently hammer every URL. One origin is primary for each target; assigned remotes add a second public IP so an outage can be confirmed without stacking load.

Who probes what

• Primary is denver by default. Change it with Shift fleet or a per-target override. If the chosen origin is offline or is the app’s own host, Denver covers until it can.
• Cloud Linux agents (GCE boxes that already report in: addo-prod-web, case-worker-1, rezamp-app-web, rezamp-app-batch) may be primary or take extra jobs. On-prem GPU boxes and Windows hosts do not probe apps.
• One extra origin per target by default (monitor.remote_probe_slots). Remaining origins are hashed so they are spread out, not every agent hitting every app. Denver is in that pool when it is not already primary.
• Never self-probe. The box that hosts an app is never asked to GET its own health URL (FastHOA is not probed from addo-prod-web). Home host comes from the target map, or from investigate.host on the last snapshot.
• Check now always runs from this monitor, even if primary has been shifted.

When a target is down

• A single origin failing is unconfirmed (shown degraded): “down from Denver only (1/2 probe IPs)”.
• The assigned remote is then asked to confirm. If it also fails, the target is down and an incident can open.
• If Denver fails and the remote succeeds, that is a split view (path problem, not a dead app). The target stays up. Origin badges still show who failed.
• Until a second origin is actually alive, the old rule still applies: monitor.fail_threshold consecutive Denver failures (default 3) open an incident so you are not stuck unable to alert.

Cadence and load

• Healthy: every monitor.check_interval_seconds (default 300). Two origins → about 24 GETs/hour per app.
• After a failure: that target retries every monitor.retry_interval_seconds (default 60) until it recovers, still only from its primary plus assigned remotes.
• Each probe is one GET or HEAD, timeout from the target form (default 10s). Health JSON should stay well under a second and a few kilobytes.
• Target list and target detail show Probe load: planned GET/hour, who is on the schedule, last hour’s actual count, and an estimate of daily bytes.

Settings that change this: monitor.probe_primary (fleet default; Shift fleet on Targets), monitor.check_interval_seconds, monitor.retry_interval_seconds, monitor.fail_threshold, monitor.confirm_origins (default 2), monitor.remote_probe_slots (default 1). The next success from the schedule resolves an open incident.

The probe token

UpDawg Monitor holds one fleet token (Settings → Monitor → Issue token). Every watched app stores the same value as UPDAWG_HEALTH_TOKEN in its .env. The checker sends it on every probe:

Authorization: Bearer <UPDAWG_HEALTH_TOKEN>
User-Agent: UpDawgMonitor/1.0

• Copy the token from the monitor UI. Do not commit it. Do not put it in this public doc.
• Compare with hmac.compare_digest. Missing or wrong token → 401 with no diagnostic body.
• Rotating the token in the monitor 401s every app until that app’s .env is updated and the process restarted.

Two endpoints

Existing /healthz or /api/health paths are fine for liveness. The monitor target URL should be the token-gated path once you ship it.

Classification on the watcher:

Healthy interval, down retry, and fail count live on Settings → Monitor. A target may override interval or fail count; 0 means inherit.

Token-gated payload

{
  "status": "ok",
  "app": "your-app-slug",
  "checks": {
    "http": "ok",
    "database": "ok"
  },
  "system": {
    "overall": "healthy",
    "queues": {"jobs": 0}
  },
  "investigate": {
    "summary": "What this app is, in one or two sentences.",
    "host": "addo-primary-den",
    "runbook": "What degraded means here, and the smallest safe fix.",
    "signals": {"jobs": "Background job queue depth"}
  }
}

• 200 when every required check is ok.
• 503 plus "status":"error" when a required dependency is out. Put the failing check in checks (e.g. "database":"error"). A short detail is allowed (“database unreachable”).
• 401 when the Bearer token is missing or wrong. Do not include checks.
• Optional system object for app-specific snapshot (queues, workers, fleet). The monitor stores it with each check and shows it on the target page. Keep it compact. Soft problems can stay HTTP 200; put them in checks as degraded or error.
• Optional investigate object (top-level, or under system) so Investigate knows what this app is. summary (what it is), host (monitor host slug), runbook (what to look at / typical fix), signals (what named checks mean). Compact, no secrets. The model reads this when you click Investigate on the target.
• Do not put secrets, stack traces, connection strings, or internal hostnames in the JSON.
• GET, fast, no side effects. Aim for well under a second.

In the monitor target form: expected status 200, JSON key status, JSON value ok.

Drop-in (Flask)

Copy snippets/flask_health_monitor.py from the UpDawg Monitor repo, or paste:

import hmac, os
from flask import jsonify, request

def _authorized():
    expected = (os.environ.get("UPDAWG_HEALTH_TOKEN") or "").strip()
    auth = request.headers.get("Authorization") or ""
    offered = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
    return bool(expected) and bool(offered) and hmac.compare_digest(offered, expected)

@app.route("/health")
def health():
    return jsonify(status="ok", app="your-slug")

@app.route("/health/monitor")
def health_monitor():
    if not _authorized():
        return jsonify(status="error", detail="unauthorized"), 401
    # optional: SELECT 1; on failure return 503 + checks.database=error
    return jsonify(status="ok", app="your-slug", checks={"http": "ok"})

.env (never committed):

UPDAWG_HEALTH_TOKEN=<paste from monitor Settings → Monitor>

Nginx must proxy /health and /health/monitor to the app. A location that only serves the marketing site will make the watcher see HTML 200s.

Paste into each project’s CLAUDE.md

## UpDawg Monitor health

Public liveness: GET /health → {"status":"ok","app":"<slug>"} (no secrets, no auth).
Watcher probe: GET /health/monitor
  Authorization: Bearer $UPDAWG_HEALTH_TOKEN
  401 if missing/wrong. 200 {"status":"ok","app":"...","checks":{...}} or 503.
Token is issued at https://monitor.addo.io (Settings → Monitor). Same value
in this app's .env as UPDAWG_HEALTH_TOKEN. Do not log it or commit it.
Do not put stack traces or secrets in the JSON.
After deploy, point the monitor target URL at https://<host>/health/monitor
(expected JSON status=ok). Docs: https://monitor.addo.io/docs/

Per-project rollout

For each app: add the token to .env on the box (not git), ship /health/monitor, then edit the target URL in this monitor. Keep public liveness on the old path so load balancers do not break.

1. Issue (or copy) the token at Settings → Monitor (admin).
2. Set UPDAWG_HEALTH_TOKEN on that app’s production host. Restart gunicorn/systemd.
3. Paste the CLAUDE.md block above into that repo.
4. Ship the token-gated route. curl -H 'Authorization: Bearer …' https://host/health/monitor should return status=ok.
5. In this monitor: Targets → that app → set URL to the new path → Check now.

What “healthy” should mean

• Liveness (public /health): the process is up. Enough for a marketing site.
• Readiness (token /health/monitor): also SELECT 1 against the app database (and Redis if the app cannot run without it).
• Workers: a separate target. Same Bearer token. Fail if the last heartbeat is older than ~90 seconds. A dead worker must not look like a dead site, and vice versa.

Edge cases

• TLS only. Production URLs are https://. The probe follows redirects; a 301 to login is down.
• Do not require cookies on either health path.
• HEAD is allowed on public liveness if you cannot return JSON. Prefer GET on /health/monitor so the watcher can assert status=ok.
• This monitor’s own liveness stays GET /health → {"status":"ok","app":"updawg-monitor"}, unauthenticated, so a peer watcher can still ping it.

On-prem and cloud host attributes

Application health is not the same as server health. UpDawg Monitor pulls CPU, memory, disk, thermal, GPU util, pending security updates, reboot-required, Docker, Ollama placement, and a short log snapshot from each box over Tailscale SSH (timer on addo-primary-den). Set ssh_target on the host (for example ubuntu@addo-standby or local for the monitor box itself).

If a machine cannot accept SSH from Denver, use its install command to enroll the push agent. The ingest token is per-host and is not UPDAWG_HEALTH_TOKEN. On Windows, enrollment installs a machine-wide Scheduled Task as SYSTEM. It reports CPU, RAM, disk, pending Windows updates, and reboot state every five minutes.

HOST_TOKEN=... MONITOR_URL=https://monitor.addo.io \
  python scripts/host_agent.py

irm https://monitor.addo.io/enroll/ONE_TIME_CODE | iex

A host is stale if no sample arrives within its stale window (default 180s). Family Windows hosts use a one-day stale window so a sleeping PC is not an outage.

Investigate on a host or target page sends that snapshot to a frontier model (Settings → Monitor API key). For apps, include an investigate object on /health/monitor so the model has runbook context instead of guessing. The reply is stored as advice: what is going on, why, a proposed fix, and a caution. Copy session prompt puts that advice on the clipboard for a coding session on that app. Apply on host queues an allowlisted systemctl restart or docker restart after you confirm; it does not run free-form shell. When the host or target is healthy again, that investigation is cleared and any open incident is marked resolved.

Maintenance can queue security-only updates, upgrade all, one package, or a reboot. Linux uses apt; Windows uses the native Windows Update API. Confirm each click. Upgrade all runs apt full-upgrade (including phased and held-back packages) and reboots automatically when the host requires it. Security-only updates never reboot. A scheduled reboot waits until the selected UTC time. Cancel a pending job before it starts.