Health API setup

The probe token

UpDawg Monitor holds one fleet token (Settings → Monitor → Issue token). Every watched app stores the same value as UPDAWG_HEALTH_TOKEN in its .env. The checker sends it on every probe:

Authorization: Bearer <UPDAWG_HEALTH_TOKEN>
User-Agent: UpDawgMonitor/1.0

• Copy the token from the monitor UI. Do not commit it. Do not put it in this public doc.
• Compare with hmac.compare_digest. Missing or wrong token → 401 with no diagnostic body.
• Rotating the token in the monitor 401s every app until that app’s .env is updated and the process restarted.

Two endpoints

Existing /healthz or /api/health paths are fine for liveness. The monitor target URL should be the token-gated path once you ship it.

Classification on the watcher:

An incident opens after fail_threshold consecutive failures (default 2). The next success resolves it and can send a recovery alert.

Token-gated payload

{
  "status": "ok",
  "app": "your-app-slug",
  "checks": {
    "http": "ok",
    "database": "ok"
  }
}

• 200 when every required check is ok.
• 503 plus "status":"error" when a required dependency is out. Put the failing check in checks (e.g. "database":"error"). A short detail is allowed (“database unreachable”).
• 401 when the Bearer token is missing or wrong. Do not include checks.
• Do not put secrets, stack traces, connection strings, or internal hostnames in the JSON.
• GET, fast, no side effects. Aim for well under a second.

In the monitor target form: expected status 200, JSON key status, JSON value ok.

Drop-in (Flask)

Copy snippets/flask_health_monitor.py from the UpDawg Monitor repo, or paste:

import hmac, os
from flask import jsonify, request

def _authorized():
    expected = (os.environ.get("UPDAWG_HEALTH_TOKEN") or "").strip()
    auth = request.headers.get("Authorization") or ""
    offered = auth[7:].strip() if auth.lower().startswith("bearer ") else ""
    return bool(expected) and bool(offered) and hmac.compare_digest(offered, expected)

@app.route("/health")
def health():
    return jsonify(status="ok", app="your-slug")

@app.route("/health/monitor")
def health_monitor():
    if not _authorized():
        return jsonify(status="error", detail="unauthorized"), 401
    # optional: SELECT 1; on failure return 503 + checks.database=error
    return jsonify(status="ok", app="your-slug", checks={"http": "ok"})

.env (never committed):

UPDAWG_HEALTH_TOKEN=<paste from monitor Settings → Monitor>

Nginx must proxy /health and /health/monitor to the app. A location that only serves the marketing site will make the watcher see HTML 200s.

Paste into each project’s CLAUDE.md

## UpDawg Monitor health

Public liveness: GET /health → {"status":"ok","app":"<slug>"} (no secrets, no auth).
Watcher probe: GET /health/monitor
  Authorization: Bearer $UPDAWG_HEALTH_TOKEN
  401 if missing/wrong. 200 {"status":"ok","app":"...","checks":{...}} or 503.
Token is issued at https://monitor.addo.io (Settings → Monitor). Same value
in this app's .env as UPDAWG_HEALTH_TOKEN. Do not log it or commit it.
Do not put stack traces or secrets in the JSON.
After deploy, point the monitor target URL at https://<host>/health/monitor
(expected JSON status=ok). Docs: https://monitor.addo.io/docs/

Per-project rollout

For each app: add the token to .env on the box (not git), ship /health/monitor, then edit the target URL in this monitor. Keep public liveness on the old path so load balancers do not break.

1. Issue (or copy) the token at Settings → Monitor (admin).
2. Set UPDAWG_HEALTH_TOKEN on that app’s production host. Restart gunicorn/systemd.
3. Paste the CLAUDE.md block above into that repo.
4. Ship the token-gated route. curl -H 'Authorization: Bearer …' https://host/health/monitor should return status=ok.
5. In this monitor: Targets → that app → set URL to the new path → Check now.

What “healthy” should mean

• Liveness (public /health): the process is up. Enough for a marketing site.
• Readiness (token /health/monitor): also SELECT 1 against the app database (and Redis if the app cannot run without it).
• Workers: a separate target. Same Bearer token. Fail if the last heartbeat is older than ~90 seconds. A dead worker must not look like a dead site, and vice versa.

Edge cases

• TLS only. Production URLs are https://. The probe follows redirects; a 301 to login is down.
• Do not require cookies on either health path.
• HEAD is allowed on public liveness if you cannot return JSON. Prefer GET on /health/monitor so the watcher can assert status=ok.
• This monitor’s own liveness stays GET /health → {"status":"ok","app":"updawg-monitor"}, unauthenticated, so a peer watcher can still ping it.

On-prem and cloud host attributes

Application health is not the same as server health. UpDawg Monitor pulls CPU, memory, disk, thermal, GPU, pending security updates, and reboot-required from each box over Tailscale SSH (timer on addo-primary-den). Set ssh_target on the host (for example ubuntu@addo-standby or local for the monitor box itself).

If a machine cannot accept SSH from Denver, issue an ingest token and run the push agent instead. That token is per-host and is not UPDAWG_HEALTH_TOKEN.

HOST_TOKEN=... MONITOR_URL=https://monitor.addo.io \
  python scripts/host_agent.py

A host is stale if no sample arrives within its stale window (default 180s).