Logline: design and phase-one plan

Working name only. "Logline" is a placeholder; rename before anything public. Do not use "Splunk" or "SPL" in the product name. "SPL-compatible" as a descriptor is fine, like "S3-compatible".

Status: draft, 2026-11-05. Nothing here is built yet.

1. Goal#

A single-host centralized logging platform for Ubuntu that:

• accepts logs over the common transports (syslog TCP/UDP, Splunk HEC, HTTP, files, journald, OTLP),
• parses and indexes them with Splunk-style sourcetypes and CIM field normalization,
• is searched with real SPL syntax, not an SPL-flavored dialect,
• installs from one binary with no hand-holding,
• monitors its own health aggressively and lets the operator restart individual services from the UI or CLI,
• treats security, secrets and SSO as first-class from day one,
• is free, open source, and built from reliable open-source pieces rather than reinventing storage or ingestion.

Non-goals (for now)#

• Multi-node clustering, search head pooling, HA. Single host only. Design so it is not impossible later, but do not spend time on it.
• The proprietary Splunk-to-Splunk forwarder protocol (S2S, port 9997). Nobody open implements it. Agents are Vector or Fluent Bit.
• Data models, tstats, summary indexing, SPL2, the Splunk app framework.
• Being a full SIEM. Detection content (Sigma rules) should run on it, but case management, SOAR, and threat intel are out of scope.

2. Premises to get right#

• SPL is proprietary. Splunk has never released SPL under an open license. Reimplementing a language's syntax is generally defensible (Google v. Oracle), but never copy Splunk documentation text, and keep Splunk trademarks out of the name and branding. Using Splunk's own software to build or test the engine is a separate, contractual risk: see 5.5. The rules for contributors are in CONTRIBUTING.md, and the points still open are under "Questions for counsel before going public" in section 13.
• CIM is free, not open source. The Splunk Common Information Model add-on is a free download under Splunk's terms. The thing worth copying is the field vocabulary (src, dest, user, action, app, the Authentication / Network Traffic / Web / Change / Endpoint models). Field names and schemas are not meaningfully protectable, and using them buys compatibility with Sigma rules and existing SPL muscle memory. Do not download, install or vendor the add-on for this project; the vocabulary comes from public documentation, in our own words. If a truly open schema matters later, OCSF (Apache-2.0, Splunk is a contributor) is the one to map to.
• No maintained open SPL engine exists. Teragrep has the most complete grammar but is AGPL and runs on Spark. Everything else is a partial parser or abandoned. The SPL compiler is the project; everything else is assembly.

3. Architecture#

Three off-the-shelf binaries plus one we write. All four run as systemd services on one host.

agents / apps / devices
        |
        v
+-------------------+      +---------------------+
|  Vector            | ---> |  ClickHouse          |
|  (ingest, parse,   |      |  (storage, SQL,      |
|   HEC/syslog/HTTP) |      |   text index, TTL)   |
+-------------------+      +---------------------+
        ^                            ^
        | generates config,          | SQL
        | reloads, scrapes metrics   |
        |                            |
+------------------------------------------------+
|  logline (Go, single binary)                    |
|  - SPL parser, planner, streaming executor      |
|  - web UI + REST API (Splunk-shaped jobs API)   |
|  - input/sourcetype/CIM management              |
|  - health, self-metrics, service control        |
|  - auth: local, LDAP, OIDC (Okta, Entra)        |
|  - installer, doctor, upgrade                   |
+------------------------------------------------+

3.1 Why these pieces#

Resource baseline to design for: 2 vCPU, 4 GB RAM, ClickHouse taking roughly 1-2 GB of it. Document a sizing table in the installer.

3.2 Processes and ports#

Nothing but Vector and the engine is reachable from the network. ClickHouse never listens on a non-loopback interface.

4. Data model#

ClickHouse table per index is unnecessary. One events table, partitioned by day, with index as a low-cardinality column, plus per-index TTL via a materialized ttl_at column.

CREATE TABLE events (
    _time        DateTime64(3, 'UTC'),
    _indextime   DateTime64(3, 'UTC'),
    index        LowCardinality(String),
    host         LowCardinality(String),
    source       String,
    sourcetype   LowCardinality(String),
    _raw         String CODEC(ZSTD(3)),
    fields       JSON,             -- index-time extracted fields, CIM names where mapped
    _id          UUID,
    ttl_at       DateTime MATERIALIZED ...,   -- computed from index retention
    INDEX raw_text _raw TYPE text(tokenizer = 'default') GRANULARITY 1,
    INDEX host_bf host TYPE bloom_filter GRANULARITY 4
) ENGINE = MergeTree
PARTITION BY toDate(_time)
ORDER BY (index, sourcetype, _time)
TTL ttl_at;

Principles:

• _raw is sacred. Index-time parsing adds fields; it never rewrites the raw event.
• Schema-on-read is what makes it feel like Splunk. Search-time extraction (rex, automatic key=value, sourcetype regex fields, lookups, calculated fields, aliases) runs in the engine over _raw and fields. Simple extractions push down to ClickHouse extract / extractAllGroups / JSONExtract.
• Index-time parsing (in Vector, driven by sourcetype definitions) does: line breaking, timestamp extraction and time zone, charset, structured decode (JSON, CEF, LEEF, key=value, CSV, syslog RFC3164/5424), and CIM field mapping. That is enough to make stats on CIM fields fast without a full Splunk props/transforms clone.
• Metrics (host and platform health) go in a separate metrics table and are queryable as index=_metrics. Platform logs go to index=_internal. Audit events go to index=_audit. Exactly like Splunk, the platform is just data in itself.

Retention: per index, configured in the UI. Hot on local disk, optional cold tier on S3-compatible storage via ClickHouse storage policies and TTL MOVE. Frozen = deleted, or exported to object storage as Parquet.

5. SPL engine#

5.1 Pipeline#

SPL text
  -> lexer / parser (hand-written recursive descent; grammar owned by us, Apache-2.0 or MIT)
  -> AST
  -> semantic pass (resolve time modifiers, macros, lookups, field aliases, subsearches)
  -> planner: splits the pipeline at the first non-pushable command
       prefix  -> one ClickHouse SQL query
       suffix  -> in-process streaming command chain
  -> executor: streams rows from ClickHouse through the suffix chain into the job's result store

This mirrors Splunk's own streaming vs. transforming split. Each command declares: pushable (can compile to SQL given its inputs), streaming (per-event, order-independent), transforming (needs all events), stateful-streaming (order-dependent, e.g. streamstats).

Subsearches run first, materialize to a value list or a filter, and are substituted into the outer plan, with a configurable result cap exactly like Splunk's.

5.2 Search semantics to replicate exactly#

• Implicit AND, explicit OR, NOT, parentheses, field=value, field!=value, comparison operators, wildcards *, quoted phrases, IN (...).
• Time: earliest, latest, relative time modifiers (-15m@m, @d, -1w@w1, now), absolute formats, _time is always the primary axis.
• Default fields: _time, _raw, host, source, sourcetype, index, _indextime, splunk_server (we will name ours something else but keep an alias; the alias is a question for counsel, section 13).
• Case-insensitive terms, case-sensitive field values, exactly as Splunk does it.
• Results table vs. events view: a search that ends in a transforming command returns a results table; otherwise it returns events with a field sidebar.

5.3 Phase-one command set#

Pushable to ClickHouse SQL:

search, where, fields, rename, eval (core functions below), stats, top, rare, sort, head, tail, dedup, timechart, chart, bin/bucket, table, fillnull, rex (simple cases via extract), regex, count shorthand, addtotals.

In-process streaming:

rex (full), eval (remaining functions), eventstats, streamstats, lookup, inputlookup, outputlookup, spath, transaction (phase 1b), append, join (phase 1b), convert, makemv, mvexpand, nomv, replace, reverse, strcat, transpose.

Eval functions, phase one: if, case, coalesce, null, isnull, isnotnull, len, lower, upper, trim, substr, replace, split, mvindex, mvcount, mvjoin, mvfilter, round, abs, ceil, floor, pow, sqrt, exact, tonumber, tostring, strftime, strptime, now, relative_time, match, like, cidrmatch, md5, sha1, sha256, urldecode, json_extract, spath.

Stats functions, phase one: count, dc, sum, avg, min, max, median, perc<N>, stdev, var, first, last, earliest, latest, values, list, range, mode, rate.

Explicitly later: map, foreach, multisearch, tstats, datamodel, collect, sendemail, iplocation (needs a GeoIP DB), geostats, predict, anomalydetection, cluster, kmeans.

Status: the commands above are built except tail, regex, addtotals, the count shorthand, lookup, inputlookup, outputlookup, transaction, append, join, convert, makemv, nomv, replace, reverse, strcat and transpose; docs/COMPATIBILITY.md is the matrix. Four of the in-process list turned out pushable: eventstats and streamstats compile to ClickHouse window functions, mvexpand to ARRAY JOIN, and spath to JSON functions (without a path it merges the JSON into the search-time field column). What runs in-process is the timechart and chart pivot, fillnull without a field list on events, and spath without a path once the field column is gone, and after those every built command, in an in-process form with the semantics of its SQL form: the executor has its own expression evaluator and aggregation (docs/COMPATIBILITY.md "In-process execution").

5.4 Knowledge objects#

Phase one: saved searches, scheduled searches with alert actions (webhook, email, write to _alerts index), lookups (CSV upload, editable in UI), macros, field aliases, calculated fields, tags, event types. Stored in SQLite inside the engine's data directory, not in ClickHouse, so they survive storage resets and are easy to back up.

5.5 Fidelity testing#

The single most valuable investment. Maintain a corpus of (query, fixture dataset, expected results) triples. Track a public compatibility matrix per command and function.

No Splunk software is installed, run or inspected for development or testing: no Splunk Free, developer, trial or cloud instance, no forwarder or add-on, and no Splunk account used to check how a query behaves. Expected results come from two sources only:

• Splunk's public documentation, read and then paraphrased in our own words with a link, never copied;
• our own reasoning about what a query over the fixture dataset must return.

Each corpus expected.json is produced by a real ClickHouse run and reviewed against those two sources before it is committed. Where the documentation leaves a semantic open (null handling, ordering, defaults, edge cases), the corpus records our best reading, the code comment says it is a reading, and the pull request flags it as uncertain so a reviewer can challenge it. Once a difference from Splunk is known, it goes into docs/COMPATIBILITY.md as a known deviation.

Why not differential testing against a real Splunk: Splunk's General Terms (updated May 2026, section 1.2, items (v) and (vi)) prohibit using any Splunk offering to "develop, test, troubleshoot, support, or market any software or service that competes" with Splunk, and testing it "for competitive purposes". An SPL-compatible engine competes with Splunk, so we read those terms as ruling out a free or developer license for this work too. In Splunk v. Cribl (April 2024), Cribl's reverse engineering was ruled fair use, but Cribl still lost on breaching those terms ($1 in damages): a fair-use win on copyright does not answer a contract claim. So the project stays out of the contract entirely rather than relying on fair use. Quotes checked against the linked pages on 2026-11-06. See CONTRIBUTING.md for the contributor rules and section 13 for the questions still open.

6. Inputs, parsing, CIM#

6.1 GUI-managed inputs#

The engine owns the Vector configuration. Inputs are rows in SQLite; the engine renders vector.yaml from them, validates with vector validate, writes atomically, and signals reload. A bad input never takes down ingestion because validation happens before the swap.

Status: built as internal/ingest with a CLI front end and JSON files instead of SQLite (see docs/INGESTION.md); the web UI and SQLite store come with the HTTP server. Two findings changed details here: Vector 0.61 no longer interpolates environment variables into its config by default, so secrets reach Vector through its file secret backend (secrets.json, mode 0600, SECRET[logline.<key>] references in the YAML); and Vector's HEC /raw endpoint ignores Splunk's host/index/source/sourcetype query parameters, which is one more reason for the engine to front HEC itself in phase 1b.

Input types in the UI, phase one:

Each input: enabled flag, default index, default sourcetype, host override, TLS settings, rate limit, and a live "last 100 events / events per second / last error" panel.

6.2 Sourcetypes#

A sourcetype is a parsing recipe with both index-time and search-time halves:

• Index time (compiled to VRL): line breaking, timestamp regex + format + timezone, charset, structured decoder, index-time field extractions, CIM mapping, event truncation limits.
• Search time (kept in engine): regex field extractions, automatic key=value, field aliases, calculated fields, lookups, tags, event types.

Ship a curated starter set with CIM mappings: syslog, linux_secure, linux_messages_syslog, nginx:access, apache:access, cef, json, windows:eventlog (via HEC/agent), cisco:asa, pfsense, sshd, sudo, journald, docker, kubernetes. Each starter sourcetype ships with sample events and a test that asserts the CIM fields it produces.

A "sourcetype workbench" in the UI: paste sample events, see line breaking, timestamps, extracted fields and CIM compliance in real time before saving. This is the single UI feature that will make or break adoption.

6.3 CIM compliance#

Implement the CIM field vocabulary and data model constraints as a built-in, versioned schema: for each model (Authentication, Change, Endpoint, Network Traffic, Network Sessions, Web, Email, Malware, Vulnerabilities, Alerts, Performance, Inventory), the required and recommended fields, their types, and allowed values (e.g. action in success|failure). Provide:

• a compliance report per sourcetype: which models it claims, which fields are present in what percentage of events,
• index=... | cimcheck model=Authentication style command (our extension, clearly marked as non-standard),
• tags and event types that mirror the CIM's expectations so Sigma rules converted with the Splunk backend run unchanged.

7. System health and service control#

This is a headline feature, not an afterthought. The health section of the UI has one page per concern and every number on it is also a searchable metric in index=_metrics.

Collected by the engine itself (gopsutil, no extra exporter):

• Host: CPU, load, memory, swap, per-mount disk usage and inode usage, disk I/O, network I/O, open file descriptors, clock skew against NTP, uptime, kernel, pending reboots.
• ClickHouse: from system.metrics, system.asynchronous_metrics, system.parts, system.disks, system.query_log, system.errors: parts per partition, merge backlog, memory, query latency percentiles, failed queries, disk per index, compression ratio, TTL activity.
• Vector: from its internal metrics API: events in/out per input, errors, buffer fill, backpressure, dropped events.
• Engine: search job queue depth, job latency, concurrent searches, scheduler lag, auth failures, API error rates, Go runtime stats.
• Ingest: events per second and bytes per day per index, sourcetype, host, input. Daily volume page even though there is no license, because people still need it.

Health checks (logline doctor CLI and the Health page share one implementation): each check returns ok|warn|fail, a human sentence, and a remediation. Examples: disk under threshold, ClickHouse reachable and not read-only, Vector config valid and all inputs bound, time sync, TLS certificate expiry, backup freshness, index TTL sanity, memory pressure, too many parts, last upgrade status.

Service control: the Health page and logline service restart <name> can restart logline-vector, logline-clickhouse, and the engine itself. The engine does not run as root. Restarts go through a tiny root-owned helper (logline-helper) listening on a Unix socket with a strict allowlist of actions, installed with a matching polkit rule. Every action is audited to index=_audit with who, what, when, and result. Logs of the restarted service's last 200 lines are shown next to the button.

Status: the Health page, the shared checks, the helper (logline-helper.service, a Unix socket reachable by the engine's group, allowlisted restart and logs, audited) and service status with restart and journal from the UI are implemented; see docs/HEALTH.md. Backup freshness is one of the shared checks (docs/BACKUP.md). The helper runs as root itself, so no polkit rule is needed. Per-component metrics, index=_metrics and the built-in alerts are not yet.

Built-in alerts (enabled by default, editable): disk above 80 and 90 percent, ingestion stopped for an input that normally receives data, ClickHouse read-only, search latency regression, certificate expiring in 14 days, failed backup, auth brute force.

8. Installer and operations#

One binary, logline, with subcommands. The installer is a state machine with explicit, logged, resumable steps and a rollback for each.

logline install            # interactive or --yes with flags/env
logline doctor             # health checks, prints and exits non-zero on fail
logline status             # services, ports, versions, disk, ingest rates
logline service <restart|stop|start|logs> <name>
logline upgrade            # engine and pinned ClickHouse/Vector versions
logline backup | restore
logline support-bundle     # redacted tarball for bug reports
logline user ...           # local admin management, break-glass reset

Install steps: preflight (Ubuntu 22.04/24.04, amd64/arm64, RAM, disk, free ports, systemd, time sync, existing installs) -> create system users -> download pinned, checksum-verified ClickHouse and Vector releases (or use an offline bundle for air-gapped hosts) -> write configs and systemd units with hardening (ProtectSystem=strict, NoNewPrivileges, PrivateTmp, resource limits) -> generate a local CA and server certificate, or configure ACME -> initialize ClickHouse schema and SQLite -> create the admin account, printing a one-time password once -> start services -> run doctor -> print the URL.

Every step logs to /var/log/logline/install.log and to the terminal with the exact command it ran and the exact error if it failed. "Just works" means the error message tells you what to do.

Status: install, doctor, service and uninstall are implemented (see docs/INSTALL.md). Steps are idempotent rather than individually rolled back: a re-run repairs, uninstall is the rollback. ClickHouse comes from the static tarball, Vector from the musl tarball, both pinned with checksums in internal/install/layout.go. backup and restore are implemented (see docs/BACKUP.md). Not yet: upgrade as its own command, the master key, ACME, the .deb.

Versions are pinned in the engine binary. upgrade upgrades the three components together, with a pre-upgrade backup of SQLite and config, a ClickHouse version compatibility check, and a rollback path.

Backups: SQLite plus configs nightly to a local path or S3; ClickHouse via BACKUP TABLE ... TO S3/Disk on a schedule. Restore is tested by doctor, not assumed.

Status: logline backup | restore and a nightly logline-backup.timer with retention are implemented (see docs/BACKUP.md): /etc/logline and the engine's data directory (JSON files, not SQLite) into one root-only archive with a checksummed manifest, the events table through BACKUP TABLE ... TO Disk('backups', ...), full backups only. Restore verifies everything before it writes, keeps a pre-restore copy, swaps the table and the files in and runs doctor; doctor reads the latest archive back on every run, and a freshness check is shared with the Health page. Not yet: S3, incremental ClickHouse backups, encrypted archives (they wait for the master key).

Packaging: a .deb too, so people who prefer apt can have it, but the single binary stays the primary path.

8.1 Growing storage and memory without reinstalling#

Storage and memory must be expandable by an operator who does not know ClickHouse. Design rules:

• Everything under one data root, /var/lib/logline, with ClickHouse data, SQLite, lookups, buffers and backups in named subdirectories. Mounting a bigger volume at that path is the simplest expansion and the installer documents it.
• Add a disk, not a migration. logline storage add-disk /mnt/data2 registers a new ClickHouse disk, adds it to the default storage policy as a JBOD volume, and new parts start landing there. Existing parts are rebalanced in the background with ALTER TABLE ... MOVE PART under doctor supervision. No downtime, no reinstall. storage remove-disk drains a disk first and refuses while data remains.
• Tiers are a policy edit. Hot local disk, warm second disk, cold S3-compatible bucket. Each is a ClickHouse storage policy volume and the Storage page shows per-index bytes per tier plus a TTL slider per index for hot-to-warm and warm-to-cold moves. Adding S3 is a form: endpoint, bucket, credentials, test button, then a background move.
• Move the data root. logline storage move /new/path stops services, rsyncs with verification, updates configs, restarts, and verifies with doctor. Rollback keeps the old copy until the operator confirms.
• Memory is derived, not hand-edited. ClickHouse max_server_memory_usage, mark and uncompressed caches, merge limits, Vector buffer sizes and the engine's search concurrency are all computed from host RAM by one tuning function. After adding RAM, logline tune (also run automatically on boot and shown as a doctor warning when host RAM changed) recomputes and applies them with a restart window the operator chooses. The Health page shows the current budget split: ClickHouse, Vector, engine, OS reserve.
• Resource profiles: small (2-4 GB), medium (8-16 GB), large (32 GB+), selectable at install and switchable later. Profiles set the same tunables plus search concurrency and query memory caps so a 2 GB host stays alive and a 64 GB host actually uses its RAM. Status: install.Tune derives the search limits with the rest (docs/SEARCH-LIMITS.md); logline tune and recomputing on a RAM change are not yet.
• Capacity forecasting. From daily ingest volume and compression ratio per index, the Storage page predicts days until each tier is full, and doctor warns at 30 and 14 days. That is what tells the operator to add a disk before it hurts.
• Never block on full disk silently. When free space drops below the floor, ingestion applies backpressure to Vector's disk buffers and raises an alert rather than letting ClickHouse go read-only unexpectedly.

9. Self-observability#

• Engine logs are structured JSON with request IDs, written to stdout for journald and also ingested into index=_internal.
• Vector and ClickHouse logs are tailed into index=_internal with proper sourcetypes.
• Every search job records its SPL, plan, SQL, timing per stage, rows per stage, and errors to index=_audit and to the job detail page (an "inspect job" view).
• support-bundle collects doctor output, recent _internal, configs with secrets redacted, versions, and system facts.
• Panics and fatal errors are caught, logged with stack traces, and surfaced in the UI as a banner with a "copy bug report" button. Opt-in crash reporting only; never phone home by default.

Status: implemented as docs/SELF-LOGGING.md describes: JSON engine and helper logs with request ids (the same id on the audit line and the X-Request-Id header), panics caught and logged with the stack, four installer-created inputs feeding index=_internal (engine, helper, Vector, ClickHouse, with sourcetypes for each) and index=_audit, searches that name no index skipping both, and a self-logging check in doctor and the Health page. The search record has SPL, SQL, timing, rows and errors but not yet per-stage figures; the job detail page, support-bundle and the panic banner are not yet.

10. Security#

• Secrets: one master key, generated at install, stored at /etc/logline/master.key mode 0400 owned by root, handed to the engine at start through systemd LoadCredential. Every stored secret (HEC tokens, LDAP bind password, OIDC client secrets, S3 keys, SMTP) is encrypted with it (XChaCha20-Poly1305) in SQLite. Secrets never appear in logs, API responses, or support bundles; a redaction layer sits in the logger.
• Passwords: argon2id. Local accounts exist for break-glass and initial setup only; encourage SSO.
• TLS everywhere external. Local CA generated at install, with a clearly documented way to install the CA on clients, or ACME for public hosts. Internal ClickHouse traffic is loopback only.
• Encryption at rest: optional ClickHouse encrypted disk (AES-256), key derived from the master key. SQLite via SQLCipher or file-level encryption. Off by default on first release, on by a flag, documented.
• API tokens: hashed at rest, scoped, expiring, revocable, shown once.
• Audit log of every login, token use, config change, service control action, search run, and data export, in index=_audit, non-deletable from the UI.
• RBAC: roles with capabilities (admin, power, user, read-only per index). Index-level access control enforced in the planner, not the UI, so the API cannot bypass it.
• Web: CSRF tokens, strict CSP, secure cookies, login rate limiting and lockout, no inline scripts, dependency pinning.
• Services run unprivileged with systemd hardening. The only root code is logline-helper with its action allowlist.
• Supply chain: reproducible builds, signed releases (cosign), SBOM published with each release, pinned and checksum-verified ClickHouse and Vector downloads.

11. SSO and identity#

• OIDC covers Okta and Microsoft Entra. One implementation, two presets with the right discovery URLs and claim names. Library: coreos/go-oidc. PKCE, state and nonce checks, group claims to roles mapping, just-in-time user provisioning.
• LDAP / Active Directory: go-ldap. Bind-as-service-account search then user bind, or direct bind with a DN template. StartTLS or LDAPS, CA pinning. Group membership (including nested groups via Active Directory's LDAP_MATCHING_RULE_IN_CHAIN) mapped to roles.
• SAML later if someone needs it; most Okta and Entra tenants are fine with OIDC.
• Mapping rules: group -> role with a default role for unmapped users, and an option to deny unmapped users. Test button in the UI that performs a real bind or discovery fetch and shows the result.
• Sessions: server-side, revocable, idle and absolute timeouts. Local admin always works even if the IdP is down.

12. Vulnerability management#

Two different things hide under this phrase. Scope them separately.

12.1 Vulnerabilities in the platform itself (phase one)#

• Each release ships an SBOM (CycloneDX) for the engine, plus the pinned versions of ClickHouse and Vector.
• The engine checks its SBOM and component versions against OSV.dev (and the GitHub Advisory Database) on a schedule, offline-tolerant, with the feed cached. No telemetry about the install is sent beyond package names and versions; document exactly what is sent and provide an air-gapped manual import.
• A Vulnerabilities page lists findings: component, CVE, CVSS, exploitability where known (EPSS, CISA KEV flag), fixed-in version, and one of three actions per finding:
  • Patch: if a release containing the fix is available, one click runs upgrade for that component.
  • Accept risk: requires a justification, an owner, and an expiry date; shows up again when it expires; appears in the audit log and in doctor output as a warning, never silently hidden.
  • Mitigate: record a compensating control with the same metadata.
• A risk summary: counts by severity, oldest unaddressed, accepted-risk register, and a plain-language "what this means" paragraph per finding generated from the advisory data.

12.2 Vulnerabilities in the user's environment (phase two)#

Ingest scanner output (Greenbone/OpenVAS, Trivy, Grype, Nessus, Qualys) as CIM Vulnerabilities data model events, then provide the same page over that data: per asset, per CVE, status tracking, accept-risk register. This is a SIEM feature and should be built as an app on top of the platform once the platform is solid, not inside it.

13. Things not yet in the list that need decisions#

• Project license. Apache-2.0 or MIT keeps it adoptable. AGPL prevents a cloud vendor from reselling it but scares contributors and makes the Teragrep grammar unusable anyway. Recommendation: Apache-2.0.
• Name, before the repository goes public.
• Agent story. Ship a documented Vector config generator ("add a host" wizard produces a ready vector.yaml and install one-liner for Linux and Windows). Windows Event Log needs an agent that speaks HEC; Vector has no Windows event log source, so recommend Fluent Bit or winlogbeat-style agents there and document it.
• Backpressure and durability. Vector disk buffers on, sized in the installer, so a ClickHouse restart drops nothing. Dead-letter index for events that fail parsing rather than silently dropping them.
• Timestamps and time zones. The number one source of wrong search results. Per-sourcetype timezone, per-input default, and a "future event" / "too old" guard that routes to the dead-letter index with a reason.
• Multiline and line breaking. Stack traces. Solve in the sourcetype workbench with explicit rules and tests.
• Search concurrency and quotas. Per-user concurrent search limits, job TTL, result size caps, query timeout, memory limits in ClickHouse per query. Otherwise one bad search takes the host down. Status: built as docs/SEARCH-LIMITS.md describes (#12): searches run as a ClickHouse account whose profile bounds time, memory, rows and bytes read, result size and threads, pinned by constraints; every search has a query id the engine kills on timeout, cancel or a client that went away; a global and per-account cap on running searches answers 429 with Retry-After; results tables are capped in rows and bytes. All derived from host RAM by the tuning function, with overrides in engine.local.env. The search jobs API (docs/JOBS.md, /api/v1/jobs) queues searches under the same caps in submission order, keeps results for a TTL and is described in OpenAPI 3.1 at /api/v1/openapi.json; it is the native API, not the Splunk-shaped one, which waits for counsel (question 3 below). Not yet: quotas over time, logline tune, jobs that outlive the engine.
• Dashboards. Even minimal dashboards (saved search panels on a grid) matter a lot to the people who love Splunk. Phase 1b. Status: built in phase 1 as docs/DASHBOARDS.md describes: base searches defined once and extended by panels, a 12-column grid of draggable, resizable panels drawn with ECharts (pie, bar, line, area, table, gauge, single value), a panel editor with a live preview, a server-side JSON store with validation and a version check on save, audited. Inputs (text boxes and dropdowns, fixed or fed by a search) feed $tokens$ into the searches, a click on a chart can set one, and a dashboard exports and imports as JSON. Panels that share a base over the same range run it once as a search job and slice its result in the engine (POST /api/v1/jobs/{id}/search); a panel alone on its base, one whose commands cannot run in the engine, or the panels of a raw-events base the event cap cut, run their own search. Not yet: time and multi-select inputs, private dashboards, more visualizations.
• Export. CSV, JSON, Parquet from any search result. Scheduled exports to S3.
• Data deletion. Per-index purge, and a GDPR-style targeted delete via ClickHouse lightweight deletes, audited.
• Field sidebar performance. Splunk's sidebar shows top values for every field. Compute over a sample, not the full result, and say so.
• Upgrade safety for ClickHouse schema. Versioned migrations with a dry-run.
• Documentation and a demo dataset shipped in the binary, so logline demo loads sample data and the first search works in under a minute.
• Resource-limited mode for a 2 GB host: smaller ClickHouse caches, lower concurrency, documented.
• Time-series metrics beyond host health. Users will want mstats-like metrics search eventually. The _metrics table should be designed so that is possible.

Questions for counsel before going public#

Nothing below is settled. Until counsel answers, the project follows CONTRIBUTING.md and does not add new Splunk-named identifiers.

1. SPL syntax reimplementation. The engine reimplements SPL's syntax and observable behavior: command, option and function names, time modifiers, defaults such as sort's 10,000-row limit, and the results-versus-events split. It is built from Splunk's public documentation (paraphrased, never copied) and our own reasoning, with no Splunk software (5.5). Is that defensible as an independent implementation? This design has leaned on Google v. Oracle and on the fair-use ruling in Splunk v. Cribl. Are there limits on how closely we may follow Splunk, for example matching error messages, option lists or the structure of its reference pages? Does working from the public documentation bring in any terms of use of the site it is published on?
2. HEC path compatibility. Vector's splunk_hec source serves /services/collector, /services/collector/event and /services/collector/raw. It accepts Authorization: Splunk <token> and Splunk's payload and metadata keys (splunk_index, splunk_sourcetype, splunk_source, splunk_channel), and our tokens are UUID-shaped to match. The CLI and installer print these paths, and phase 1b has the engine fronting HEC itself (#6). Can we accept that wire format, and how may we describe it ("HEC-compatible", "accepts the Splunk HEC protocol")? These identifiers appear in internal/ingest, cmd/logline, internal/install, testdata/vector and docs/INGESTION.md; none has been renamed.
3. A future Splunk-compatible REST API. Sections 3 and 3.2 plan a "Splunk-shaped" jobs and management API, optionally on port 8089, and question 4 in section 15 asks whether Splunk SDKs and scripts should work against it. Before any of it is built: is matching REST paths, parameters and response shapes acceptable? May we test against Splunk's SDKs, and under what licenses? How should we describe it?
4. The splunk_server alias. Section 5.2 plans to keep splunk_server as an alias of our own default field so existing searches still work. It is not built yet; today the name appears only in this document. Is a field name that contains the mark acceptable as a functional identifier, or should the alias be dropped or made opt-in?
5. Final product name and a trademark search. "Logline" is a placeholder (see "Name" above). Before going public, choose a name that contains no Splunk mark and nothing confusingly similar to one. Run a clearance search on it covering registrations and common-law use in logging and observability, and check the domain and the GitHub organization.
6. Project license. The repository is Apache-2.0 today (LICENSE, and "Project license" above). Confirm it suits a project whose purpose is compatibility with a proprietary product. Decide whether contributions need a DCO or CLA, given that CONTRIBUTING.md asks contributors to confirm they used no Splunk software and copied no Splunk documentation text.
7. CIM vocabulary and starter sourcetype names (added in the same review). internal/ingest/model.go ships data model names with required and recommended field lists that follow Splunk's Common Information Model. Some starter sourcetype names, built (linux_secure) and planned in 6.2 (cisco:asa, windows:eventlog), follow names that Splunk and its add-ons use. Section 2 treats field names as not protectable. Confirm that, and confirm the lists may be compiled from public documentation without the CIM add-on.
8. Is SPL / Search Processing Language a registered Splunk mark? (USPTO search) Its status is unconfirmed. It is not on Splunk's public trademark page, which is dated 2022 and says it is not exhaustive, and Splunk's style guide uses it unmarked. Until this is answered, internal/spl, the logline spl subcommand and the .spl corpus files stay as they are.

14. Phases#

1. Phase 0, spike (weeks): SPL parser and planner for search | stats | sort | head | timechart over a hand-loaded ClickHouse table. Prove the pushdown split and the fidelity test harness. No UI, CLI only. Done 2026-11-05.
2. Phase 1, usable: installer, Vector-managed inputs for syslog/HEC/HTTP/file, starter sourcetypes with CIM mapping, phase-one command set, web UI with timeline and field sidebar, local auth, health page, doctor, self-logging, backups. In progress: Vector-managed inputs and starter sourcetypes landed first (CLI), then the search web UI and JSON API (timeline, field sidebar, events, tables, charts), then local auth (argon2id, sessions, scoped API tokens, lockout, audit log, forced first-login password change; see docs/AUTH.md), then the installer with doctor, service and uninstall, verified end to end on an Ubuntu 24.04 systemd host (see docs/INSTALL.md), then the Health page with service control through the root helper (see docs/HEALTH.md). Then self-logging: structured engine logs with request ids, the platform's own logs and audit trail in index=_internal and index=_audit (see docs/SELF-LOGGING.md). Then backups: logline backup | restore with a nightly timer, retention, a freshness check and a tested restore (see docs/BACKUP.md). Then dashboards, pulled forward from phase 1b (see docs/DASHBOARDS.md). Still to come in phase one: the rest of the command set.
3. Phase 1b: transaction, join, subsearches, saved searches and alerts, lookups, OIDC and LDAP, RBAC, platform vulnerability page, service control from UI.
4. Phase 2: cold storage tiering, environment vulnerability management on CIM data, Sigma rule runner, metrics search, .deb packaging, encrypted-at-rest by default.

15. Open questions for the owner#

1. Settled: no differential testing against Splunk. No Splunk software is installed, run or inspected for development or testing. Expected results come from public documentation, paraphrased, and our own reasoning, and uncertain semantics are flagged in pull requests. Section 5.5 gives the reasons: Splunk's General Terms section 1.2 and Splunk v. Cribl.
2. Go is the recommendation. Is that acceptable given your current Python and TypeScript stack, or should the engine be Python with the hot paths later rewritten? (Recommendation stays Go.)
3. Target first users: just you, a homelab crowd, or small businesses? It decides how much the installer must handle (ACME, SMTP, proxies).
4. Should the Splunk-shaped REST API be a goal from phase one so Splunk SDKs and scripts work, or is the native API enough initially? (Ask counsel first: section 13, question 3.)