Self-logging

The platform is data in itself. Its own logs are searched the same way as anything else, in two indexes that ordinary searches leave alone:

A search that names no index, or says index=*, skips both (as in Splunk). index=_internal, index=_audit and index=_* reach them.

index=_internal sourcetype="logline:engine" status>=500 | table _time, request_id, path, duration_ms
index=_internal sourcetype="clickhouse:server" level=Error | stats count by component
index=_internal sourcetype=vector level=ERROR
index=_audit event=login action=failure | stats count by user, src
index=_audit event=search | stats avg(details.elapsed_ms) by actor

The engine's log#

The engine and the helper log one JSON object per line (time, level, msg, then fields) to stdout, which systemd stores in the journal. On a terminal the same logger prints key=value text instead; the format follows $JOURNAL_STREAM and can be forced with LOGLINE_LOG_FORMAT=json|text.

Every HTTP request gets a 16-hex-digit request id, returned as the X-Request-Id response header and logged on the request line:

{"time":"2026-11-06T02:55:33.408Z","level":"INFO","msg":"request","request_id":"6967c97ab8cccf74",
 "method":"POST","path":"/api/search","status":200,"duration_ms":5.736,"bytes":2210,
 "ip":"127.0.0.1","user":"admin","via":"session"}

The audit line for the same action carries the same request_id, so index=_* request_id=6967c97ab8cccf74 shows the request and what it did. Static assets are not logged. Responses of 500 and up log at ERROR.

A panic in a handler is caught: the response is a 500 with {"error":"internal error","kind":"panic","request_id":...}, and the log gets an ERROR line with msg=panic, the panic value and the stack. The HTTP server's own complaints (TLS handshake errors and the like) go through the same logger at WARN.

The audit log#

audit/audit.jsonl under the engine's data directory is the record of truth: the engine appends to it before anything reaches ClickHouse, so entries survive a storage outage. A search line records the SPL, the range, the limit, the result count, whether the cap was hit, the elapsed time, the generated SQL and the commands that ran in the engine (residual). See AUTH.md for the event list.

How it gets there#

logline install adds four inputs (listed by logline inputs list, disabled or removed like any other; a removed one is not re-added on a re-run):

For that to work the installer gives logline-vector.service SupplementaryGroups=systemd-journal and VECTOR_LOG_FORMAT=json, makes /var/log/logline/clickhouse and the audit directory setgid to logline-vector so new files are group-readable, and keeps the rest of the engine's state (users, sessions, tokens) out of Vector's reach.

Journald inputs keep the writer's identity as fields: unit, process, pid, priority; source is the unit name. --units a.service,b.service on logline inputs add --type journald limits any journald input to a set of units.

The vector sourcetype parses Vector's JSON log lines and strips the quotes Vector puts around field values. clickhouse:server extracts thread, query_id, level, component and the first line of the message; a stack trace stays in _raw because the input joins continuation lines.

Checking it#

logline doctor and the Health page have a self-logging check: ok when index=_internal received something in the last 15 minutes (or Vector started under five minutes ago), warn otherwise, with the journal command to look at. Ingest volume by index on the Health page shows _internal and _audit next to the data indexes.

Not yet#

• A per-stage plan and timing in the search record, and an "inspect job" view in the UI.
• support-bundle.
• A UI banner for a caught panic with a "copy bug report" button.
• index=_metrics: host and component metrics (Vector's internal metrics, ClickHouse query latency).