Health page and service control

The Health page (#health in the UI, roles user and admin) and logline doctor run the same checks from one implementation in internal/install. The page adds what a shell cannot show at a glance: the three services with restart and logs, host resources, ClickHouse's memory and table figures, and ingest volume.

What the page shows#

• Summary: overall status (ok, warn, fail), counts, when it was checked. The page refreshes itself every 30 seconds while open.
• Services: clickhouse, vector, engine with version, unit state, start time, PID, systemd restart count and cgroup memory. Admins get Logs (the unit's last 200 journal lines) and Restart. Restarting the engine answers first and restarts a second later; the page reconnects.
• Checks: every doctor check with its status, sentence and remediation, including self-logging (is index=_internal receiving the platform's own logs; see SELF-LOGGING.md) and backup and backup timer (did last night's backup work, is the timer on; see BACKUP.md), and search limits (the limits every search runs under, and whether ClickHouse's search profile matches them; see SEARCH-LIMITS.md).
• Resources: host CPUs and load, memory, disk under the data root, ClickHouse memory used against its configured limit, the events table (rows, bytes, compression ratio, parts, day range), engine uptime and per-search event cap, Vector's disk buffer, and searches running now against the concurrency caps.
• Running searches: who runs what and for how long, with Cancel; an admin sees everyone's, anyone else their own.
• Ingestion: events indexed in the last hour and 24 hours, a per-minute chart for the last hour, and a per-index table with the last event time (_internal and _audit included).

In development (no install under /etc/logline) the page still works: the checks that need an installed host are skipped and service control is off.

The helper#

The engine runs unprivileged, so restarts and journal reads go through logline-helper.service: the engine binary run as root with logline helper, listening on /run/logline/helper.sock (root:logline, mode 0660; nobody else can connect). It accepts one JSON line per connection with three actions: ping, restart, logs. The unit name comes from an allowlist of the platform's own units; nothing received is passed to a shell. Every restart and log read is audited (service.restart, service.logs) with the actor, the target and the result.

GET  /api/health/report                  user   the whole page as JSON
POST /api/services/{name}/restart        admin  name: engine | vector | clickhouse
GET  /api/services/{name}/logs?lines=200 admin  text/plain, journal lines

Without the helper (not an installed host, or the unit down) the report says so in helper_error, the buttons are hidden and the endpoints return 503. logline doctor warns when the helper does not answer.

Not yet#

• Per-input and per-sourcetype ingest rates, Vector's internal metrics, ClickHouse query latency: they come with the self-metrics index (index=_metrics).
• Alerts on thresholds (disk, ingest stopped, certificate expiry).
• Service control from the CLI through the helper; logline service still calls systemctl directly and needs root.