Logline

Working name. Not affiliated with Splunk. "SPL-compatible" is a descriptor, like "S3-compatible".

A single-host centralized logging platform for Ubuntu, searched with SPL syntax, built on Vector for ingestion and ClickHouse for storage, with one Go binary providing the SPL engine, web UI, inputs management, health, auth and the installer.

Read DESIGN.md for the architecture, phase plan and the decisions behind it, and TODO.md for where the work stands, what is next and how to work on it. Before contributing, read CONTRIBUTING.md: no Splunk software or accounts, no copied Splunk documentation, and no Splunk marks in names or branding.

Status#

Phase zero is done: the SPL parser, planner and ClickHouse SQL compiler, verified against ClickHouse with a 32-query corpus. Phase one so far: CLI-managed inputs and sourcetypes rendered into a validated Vector configuration, end to end into ClickHouse, and a search web UI (a data sources panel, search history, timeline, field sidebar, event list, grids and charts) over a JSON API. Local accounts with argon2id passwords, sessions, API tokens, login lockout and an audit log are in. logline install turns an Ubuntu host into a running server: pinned and checksum-verified ClickHouse and Vector, hardened systemd units, a local CA, the schema and the first admin; logline doctor checks it afterwards, and the Health page shows the same checks with service status, restart and logs through a small root helper, host resources and ingest volume. The platform logs itself: the engine writes JSON lines with request ids, and its own logs, Vector's, ClickHouse's and the audit trail are searchable as index=_internal and index=_audit. Dashboards: saved grids of panels (pie, bar, line and area charts, tables, gauges, single values) over base searches and inputs, with a drag-and-resize editor and JSON export and import. Nothing here is ready for use. See docs/INSTALL.md, docs/SEARCH.md, docs/HEALTH.md, docs/DASHBOARDS.md, docs/JOBS.md, docs/SELF-LOGGING.md, docs/BACKUP.md, docs/INGESTION.md, docs/AUTH.md and docs/COMPATIBILITY.md.

Install on a host#

On Ubuntu 22.04 or 24.04 with systemd, as root:

sudo ./logline install

It checks the host, asks once, installs everything under /etc/logline, /var/lib/logline and /opt/logline, starts three services and prints the URL and a one-time admin password. Re-running it is safe. Details, layout, sizing and removal are in docs/INSTALL.md.

Layout#

cmd/logline/          CLI entry point
internal/spl/         lexer, parser, AST, relative time modifiers
internal/plan/        planner: splits a pipeline into a ClickHouse pushdown and a residual
internal/chsql/       AST -> ClickHouse SQL
internal/exec/        in-process streaming commands for the residual (rex, rename, fields, head, pivot)
internal/ch/          minimal ClickHouse HTTP client, no third-party dependencies
internal/schema/      the events table DDL
internal/ingest/      inputs, sourcetypes, CIM vocabulary, VRL compiler, Vector config renderer
internal/vector/      runs Vector (local binary or the pinned Docker image) to validate and replay
internal/search/      parse, plan, query and post-process in one call; field summaries; timeline
internal/server/      HTTP API and the embedded web UI (internal/server/ui, plain HTML/CSS/JS)
internal/auth/        local accounts, sessions, API tokens, lockout, audit log
internal/dashboard/   dashboards: the model, validation and the JSON-file store
internal/history/     each account's search history, one JSON file per account
internal/jobs/        search jobs: the queue under the concurrency caps, states and result retention
internal/install/     the installer: layout, preflight, downloads, units, TLS, doctor, uninstall
internal/health/      check/report types shared by preflight, doctor and the Health page
internal/logx/        the structured logger (JSON under systemd, text on a terminal)
internal/devdata/     deterministic fixture dataset
testdata/corpus/      one directory per query: query.spl, plan.sql (golden), expected.json
testdata/vector/      golden rendered vector.yaml

Develop#

Requires Go 1.25+ and, for running queries, a ClickHouse reachable over HTTP.

go build -o bin/logline ./cmd/logline
bin/logline spl plan 'index=web status>=500 | stats count by host | sort -count | head 5'

Start a throwaway ClickHouse, load the fixture dataset, run a search:

docker run -d --name logline-ch -p 127.0.0.1:8123:8123 -e CLICKHOUSE_PASSWORD=logline clickhouse/clickhouse-server:latest
bin/logline dev load --dsn http://default:logline@127.0.0.1:8123
bin/logline spl run --dsn http://default:logline@127.0.0.1:8123 --earliest -30d 'index=web | timechart span=1d count by status'

Define inputs and sourcetypes, generate Vector's config, and run Vector against the same ClickHouse (full walkthrough in docs/INGESTION.md):

bin/logline inputs add --type hec --id hec --port 8088
bin/logline inputs add --type syslog --id syslog --port 5514 --index os
bin/logline sourcetypes test linux_secure --file /var/log/auth.log
bin/logline vector apply --ch-endpoint http://logline-ch:8123 --ch-password logline

Start the web UI and open http://127.0.0.1:8000:

bin/logline serve --dsn http://default:logline@127.0.0.1:8123

On the first start it creates the admin account and prints a one-time password; sign in and change it. Accounts can also be managed from the terminal (logline users list|add|passwd|role|enable|disable|rm|tokens). Every login, search, and account or token change is appended to audit/audit.jsonl in the data directory. See docs/AUTH.md.

The UI is one client of the JSON API. Sign in with POST /api/auth/login, or send an API token created on the Account page as Authorization: Bearer ll_.... POST /api/search takes {"query": "...", "earliest": "-24h", "latest": "now", "limit": 1000, "timeline": true, "fields": true, "tz": "Europe/Berlin"} and returns the columns and rows, the generated SQL, the histogram buckets and the field summary. tz (UTC when left out, --tz on the CLI) is the zone earliest, latest and the time functions are read in. GET /api/health reports the ClickHouse connection; GET /api/inputs and GET /api/sourcetypes[/name] expose the ingestion configuration read-only. Programs should use the search jobs API instead: POST /api/v1/jobs submits a search that queues under the concurrency caps, GET /api/v1/jobs/{id} polls or waits on it and /results pages its rows or downloads CSV; the engine serves its OpenAPI 3.1 description at /api/v1/openapi.json (docs/JOBS.md).

Run the unit tests and, with a ClickHouse available, the integration corpus:

go test ./...
LOGLINE_CH_DSN=http://default:logline@127.0.0.1:8123 go test -tags integration ./...

License#

Apache-2.0. See LICENSE.