TODO

Where the work stands and how to pick it up. The long-form plan is DESIGN.md (section 14 has the phases); each docs/*.md page ends with a "Not yet" list for its area. Open work is tracked as GitHub issues labelled phase-1, phase-1b, phase-2; this file is the map.

State (2026-11-07)#

Phase 0 is done. Phase 1 pieces landed in this order and are committed on main. Items 1 to 6 were verified end to end as they landed; backups (7) and search limits (8) were verified on 2026-11-07 on the systemd container recipe below (a backup, the oneshot unit under its sandbox, a restore after deleting the engine directory and the events table, doctor; the ClickHouse search account and its constraints, KILL QUERY, the engine's 429, cancel and client-disconnect paths, the corpus under the search profile). Not yet on a bare Ubuntu host.

1. Vector-managed inputs and sourcetypes, CLI (docs/INGESTION.md)
2. Search API and web UI (README.md, docs/SEARCH.md, internal/server)
3. Local auth (docs/AUTH.md)
4. Installer: install, doctor, service, uninstall (docs/INSTALL.md)
5. Health page and the root helper (docs/HEALTH.md)
6. Self-logging: index=_internal, index=_audit (docs/SELF-LOGGING.md)
7. Backups: logline backup | restore, a nightly timer with retention, a freshness check, a tested restore (docs/BACKUP.md, #1)
8. Search limits: a ClickHouse search account whose profile bounds every search, KILL QUERY on timeout, cancel and disconnect, concurrency caps with 429, result caps, all derived from host RAM (docs/SEARCH-LIMITS.md, #12)
9. Dashboards, pulled forward from phase 1b: base searches, a grid of resizable panels (pie, bar, line, area, table, gauge, single value), a panel editor with live preview, saved on the server and audited; then inputs and tokens (text boxes and dropdowns, fixed or search-fed, $id$ in the searches, drilldown into an input), JSON export and import, duplicate, CSV per panel, present mode and add-to-dashboard from the search page (docs/DASHBOARDS.md); verified on the systemd container with the shipper's data
10. Search jobs API: /api/v1/jobs, asynchronous searches that queue under the concurrency caps in submission order, keep their result for a TTL, page it as JSON or CSV, cancel through the job or the running-searches endpoint, audited; the engine serves its OpenAPI 3.1 description at /api/v1/openapi.json (docs/JOBS.md, #20); verified on the systemd container

Remaining in phase 1: the rest of the command set. Then phase 1b.

Next up (phase 1)#

• Command set (#2; docs/COMPATIBILITY.md lists what is pushed): stats or eval functions beyond the DESIGN.md 5.3 lists. chart, eventstats, streamstats, fillnull, mvexpand and spath are done (#2), with chart's span and where, streamstats' reset_* and time_window, spath's array indexes and timechart's usenull and cont; the leftovers are in COMPATIBILITY.md: chart's bins, streamstats' allnum, spath's XML and {} names when it merges into the search-time fields. Each new command gets a corpus directory under testdata/corpus/ (query, golden plan.txt, expected.json from a real ClickHouse run, reviewed against the public documentation and our own reasoning, never against Splunk; see DESIGN.md 5.5). The gaps #3 listed in the existing commands are closed; what remains of them is in the known deviations there, chiefly time buckets in UTC whatever the search's time zone, and dedup's keepempty, keepevents and consecutive.
• In-process executor (internal/exec): it runs every built command after the point a pipeline leaves SQL (COMPATIBILITY.md "In-process execution"), with an expression evaluator (eval.go, strftime.go) and aggregation (agg.go) that mirror the SQL forms, and it serves the dashboards' shared base results (POST /api/v1/jobs/{id}/search). Leftover: it only sees the rows the query returned, so a transforming command after fillnull or spath on events works over at most the event cap.
• Search limits leftovers (docs/SEARCH-LIMITS.md "Not yet"): logline tune, quotas over time, fair sharing while jobs wait. Jobs leftovers (docs/JOBS.md "Not yet"): the UI on jobs, jobs that outlive the engine, progress, streaming results.

Phase 1b and later#

• transaction, join, append, lookup, subsearches (parsed, rejected by the planner).
• Saved searches and alerts; the built-in alerts in DESIGN.md section 7.
• Dashboard leftovers (docs/DASHBOARDS.md "Not yet"): time and multi-select inputs, private dashboards, more visualizations.
• OIDC (Okta, Entra) and LDAP, RBAC with per-index access (#7; the roadmap below has the roles).
• Inputs and sourcetypes editable from the UI (read-only today), the engine fronting HEC (#6; the roadmap below has the Inputs page).
• index=_metrics: host and component metrics; per-input ingest rates and the trend charts on the Health page (#5; the roadmap below).
• Self-logging leftovers: per-stage timing in the search record and an "inspect job" view, support-bundle, a UI banner for a caught panic (#4).
• Backup leftovers: S3 targets, incremental ClickHouse backups, encrypted archives (after the master key), a backups panel on the Health page (docs/BACKUP.md "Not yet").
• Installer leftovers: logline upgrade with a pre-upgrade backup, the master key and LoadCredential, TLS on HEC and syslog, ACME, a .deb, storage add-disk (#8; the roadmap below has the disk growth plan).
• Vulnerability page, Sigma rules, cold tiering (phase 2).

Roadmap (owner, 2026-11-07)#

The features the owner wants next, in their words with a first cut at what each means, each with its issue; the bullets in "Phase 1b and later" that overlap point here.

• Growing disk space without losing the datastore (#28). Today a full disk means a bigger host and a restore. Wanted: logline storage add-disk as DESIGN.md 8.1 describes it (the volume joins the storage policy as a JBOD disk, new parts land there, old parts are rebalanced in the background with MOVE PART, nothing is reinstalled), storage remove-disk that drains first, a documented way to move the whole data directory to a bigger volume with the services stopped, and a Health check that forecasts when the disk fills from the ingest rate ("full in 9 days at this rate"). #8 carries storage add-disk. Retention per index (DESIGN.md section 4) is the other half of not running out.
• Tenant configuration in git (#29). Every knowledge object and setting that is not data lives in a git repository: inputs, sourcetypes (custom, imported and the library ones a tenant changed), dashboards, saved searches, alerts, lookups, macros, users and roles (never secrets: tokens, passwords, keys and secrets.json stay out, with a check that refuses a commit holding one). Default is a local bare repository under the data directory (config.git), written by the engine on every change from the UI, CLI or API with the actor as the commit author and the audit line's request id in the message, so the audit log and the history agree; optionally mirrored to an external remote over SSH with a deploy key (logline config remote set). logline config log | diff | restore <ref> and a Configuration page that shows the history and a diff; a restore is itself a commit. Backups keep working as they are; this is the version history, not the backup.
• Scalability: 10 million events per hour (#30). About 2,800 events per second sustained, bursts above it. A load generator (logline bench, or the shipper container made parametric) feeding syslog, HEC and JSON over TCP; measured: ingest lag (_indextime minus _time), Vector buffer depth and drops, ClickHouse inserts per second, parts and merges, CPU, RAM and disk per service, search latency for the corpus under load, and 24 hours without a restart. Then tune what the numbers say (Vector batch size and buffers, ClickHouse async inserts and merge settings, the search profile) and write the sizing table into INSTALL.md. Container first, a real host after; a nightly run of the short version would catch regressions.
• A library of common sourcetypes (#31). A shipped, versioned set under internal/ingest/library/: nginx and Apache access and error logs, syslog (RFC 3164 and 5424), journald, Linux auth, Docker and Kubernetes JSON, HAProxy, PostgreSQL, MySQL, Windows event forwarding as JSON, cloud audit logs (AWS CloudTrail, GCP), each with sample events, the fields it extracts, its CIM mapping and a test that the samples parse. Importable from the UI and CLI into a tenant's sourcetypes as a copy ("imported from the library, version 3") that can be changed, with an upgrade path when the library moves. The sourcetype workbench (DESIGN.md 6.2) is where a library entry is tried against a tenant's own events.
• Dashboards page overhaul (#32). The list must say at a glance which dashboards ship with Logline for self-monitoring (ingest, search activity, health, audit) and which a tenant made. Built-ins ship as read-only with "Clone to edit", get their own section with a Logline mark, and update with the engine; tenant dashboards show owner, last change, a thumbnail or a sparkline of their first panel, tags or folders, favorites, and a search box; the list filters by all of it.
• Health page trends (#33). Memory, disk and network throughput over time (an hour, a day, a week), per service where it applies, as animated time-series charts next to the figures the page shows now; the disk chart carries the fill forecast above. This needs the _metrics index (#5): a collector in the engine sampling /proc and ClickHouse's system tables every few seconds and writing them as events, so every chart on the page is also a search anyone can run and put on a dashboard.
• Inputs page (#34). Inputs grouped by type (syslog, HEC, file, journald, TCP and UDP, the engine's own), a filter box, and a card per input with its state: enabled, listening where, events in the last hour, the newest event, and errors from Vector's log. Create, edit, enable, disable and delete from the page with a form per type that validates as the CLI does, a "send a sample" test that shows the parsed event before saving, and Apply that renders and validates the Vector configuration and shows the diff before it restarts Vector. A new input picks its sourcetype from the tenant's own or from the library. #6 is the issue; it also covers the engine fronting HEC.
• RBAC (#35). Roles with named capabilities, local or from SSO, enforced in the API and the planner and only reflected in the UI. A first set: readonly (search, view dashboards), user (plus own dashboards, own saved searches, own tokens), power (plus shared dashboards and saved searches, alerts, lookups, macros, scheduled searches, the sourcetype workbench), admin (everything: users, roles, inputs, sourcetypes, service control, settings, the configuration history). Per-index access on top: a role or a user lists the indexes it may search, enforced where the planner resolves index= so the API cannot bypass it, with _internal and _audit admin-only by default. Capabilities are a flat list (search, dashboards.write, alerts.write, inputs.write, users.write, ...) so a custom role is a name plus a set; SSO maps groups to roles (#7) with a default role and a deny-unmapped option; every role change is audited; tokens carry their owner's role at use time, never more. A Roles page for admins, and the capability matrix in docs/AUTH.md.

Working on it#

Read first: CONTRIBUTING.md, then DESIGN.md, then the doc page for the area. CONTRIBUTING.md sets the rules for SPL compatibility work: no Splunk software or accounts, no copied Splunk documentation, no Splunk marks or logos, and no reverse engineering. Conventions that are not obvious from the code:

• Go 1.26, no third-party dependencies beyond golang.org/x/crypto and golang.org/x/sys; keep it that way unless a dependency buys a lot. The UI vendors two browser libraries for the dashboards (ECharts and GridStack, released files checked in with versions, licenses and checksums in internal/server/ui/vendor/README.md); anything else in the UI is plain HTML, CSS and JS.
• go test ./... must pass; go vet ./... and gofmt -l clean. With a ClickHouse at hand, LOGLINE_CH_DSN=http://default:logline@127.0.0.1:8123 go test -tags integration ./... runs the SPL corpus. Golden files (testdata/corpus/*/plan.txt, testdata/vector/vector.yaml) are regenerated with -update.
• Sourcetype VRL is checked through real Vector: bin/logline sourcetypes test <name> --file sample.log (local binary or the pinned Docker image).
• Installer changes are verified on a real systemd host. A Docker container works: docker run -d --name logline-ubuntu --privileged --cgroupns=host --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:rw -p 127.0.0.1:8443:443 jrei/systemd-ubuntu:24.04, then apt-get install curl ca-certificates, copy in a GOOS=linux GOARCH=amd64 CGO_ENABLED=0 build and run logline install --yes. Put the two release tarballs (logline install --artifacts lists them) in /var/cache/logline first to skip the downloads. Rehearsal mode, logline install --root /tmp/r --yes, writes every file on any OS and is what the unit tests use.
• Commit messages describe what and why in full sentences; one piece of work per commit; docs and the DESIGN.md status notes move with the code.
• Nothing is released. Versions of ClickHouse and Vector are pinned with checksums in internal/install/layout.go; bump them together.

Repository rules#

• main is meant to be protected: no direct pushes, no force pushes, no deletion, and every pull request needs an approving review from the owner (.github/CODEOWNERS), with review threads resolved. The ruleset is .github/rulesets/main.json; the owner can merge their own pull requests without a second reviewer but cannot push to main directly.
• GitHub only enforces rulesets on a private repository under a paid plan (GitHub Pro), or on a public one. Until the repository is public or the account is upgraded the file is documentation; to enable it, import it under Settings, Rules, Rulesets, "Import a ruleset", or POST /repos/Wimanaconda/logline/rulesets with the file as the body.
• Work on a branch and open a pull request; the owner's review is requested automatically through CODEOWNERS either way.