Ingestion: inputs, sourcetypes and the generated Vector config

The engine owns Vector's configuration. Inputs and sourcetypes are small JSON documents in the data directory (./data or $LOGLINE_DATA); logline vector apply renders them into one vector.yaml, validates it with vector validate, and only then swaps it into place. Vector reloads on its own with --watch-config. A bad input therefore never takes ingestion down. Adding, removing, enabling or disabling an input changes nothing until the next logline vector apply; the commands say so.

data/
  inputs.json           every input
  sourcetypes/*.json    custom sourcetypes (override builtins of the same name)
  vector/vector.yaml    generated; do not edit
  vector/secrets.json   ClickHouse password and HEC tokens, mode 0600

Inputs#

Every input takes --index, --sourcetype, --host and --source defaults. Events that already carry a value keep it: HEC payloads set their own index, sourcetype, source and host; syslog headers set host and _time.

A network input (syslog, tcp, udp) names each event after the sender unless the event names itself: by default host is the sender's IP address, --host-from dns looks the address up (a failed lookup keeps the address), and --host fixes one name for everything the input receives. Send JSON with a host key, or syslog with a hostname in the header, and that wins either way. The sender's port is not kept.

logline inputs add --type hec --id hec --port 8088
logline inputs add --type syslog --id syslog --port 5514 --index os
logline inputs add --type syslog --id secure --port 5516 --protocol tcp --index os --sourcetype linux_secure
logline inputs add --type tcp --id app-json --port 5515 --index app --sourcetype json
logline inputs add --type tcp --id nginx --port 5517 --index web --sourcetype nginx:access --host-from dns
logline inputs list
logline inputs disable syslog

Sourcetypes#

A sourcetype is an index-time parsing recipe compiled to VRL. The declarative parts run in this order:

1. decoder on _raw: json, syslog, kv, cef or regex (with regex, named groups become fields).
2. extract: more regexes with named groups, merged into the fields.
3. numeric: fields converted to numbers.
4. timestamp: field plus a strftime format (empty means best effort: ISO 8601, YYYY-MM-DD HH:MM:SS, epoch seconds or milliseconds) and an optional timezone for formats without a zone.
5. cim.map (CIM field <- extracted field) and cim.static.
6. vrl: appended verbatim for anything the above cannot express.

Builtins: syslog, linux_secure (sshd and sudo, Authentication model), nginx:access and apache:access (Web model), json, kv, cef, journald, and the platform's own logline:engine, logline:audit (login lines in the Authentication model), vector and clickhouse:server (see SELF-LOGGING.md). A custom file with the same name overrides a builtin.

logline sourcetypes list
logline sourcetypes show linux_secure        # the JSON definition
logline sourcetypes vrl linux_secure         # the VRL it compiles to
logline sourcetypes add --file my_app.json
logline sourcetypes test nginx:access --file sample.log --host web-01

sourcetypes test replays sample lines through the same three VRL stages Vector runs in production (input tagging, sourcetype parsing, finalize) and prints the fields each event ends up with and whether the CIM models the sourcetype claims are satisfied. It needs a vector binary on the PATH or Docker, in which case the pinned Vector image is used.

The generated pipeline#

in_<input>  -> tag_<input>  (defaults, _raw guarantee, HEC metadata)
            -> route        (by sourcetype)
            -> parse_<sourcetype>
            -> finalize     (builds the events row: _time, _indextime, index,
                             host, source, sourcetype, _raw, fields)
            -> clickhouse sink (basic auth, 1 s batches, disk buffer)

Secrets never appear in vector.yaml. Vector 0.61 disabled environment variable interpolation in config files by default, so the config references SECRET[logline.ch_password] and SECRET[logline.hec_<id>], resolved by Vector's file secret backend from secrets.json next to the config. --vector-dir (default /etc/vector) is where Vector sees that directory; in Docker, mount the output directory there.

Running Vector in Docker for development#

docker network create logline
docker network connect logline logline-ch
logline vector apply --ch-endpoint http://logline-ch:8123 --ch-password logline --api 0.0.0.0:8686
docker run -d --name logline-vector --network logline \
  -p 127.0.0.1:8088:8088 -p 127.0.0.1:5514:5514/udp -p 127.0.0.1:5515:5515 -p 127.0.0.1:5516:5516 \
  -v "$PWD/data/vector:/etc/vector" timberio/vector:0.61.0-alpine \
  --config /etc/vector/vector.yaml --watch-config --watch-config-method poll

--watch-config-method poll matters on Windows and macOS, where inotify events do not cross the bind mount; on a Linux host the default works.

Send something:

TOKEN=$(sed -n 's/.*"hec_hec": "\([^"]*\)".*/\1/p' data/vector/secrets.json)
curl -H "Authorization: Splunk $TOKEN" -d '{"event":{"level":"error","msg":"db timeout"},"sourcetype":"json","index":"app"}' http://127.0.0.1:8088/services/collector/event
echo '<13>Nov  5 14:22:09 web-01 sshd[1240]: Accepted password for alice from 10.0.0.5 port 51234 ssh2' | nc -u -w1 127.0.0.1 5514
logline spl run --dsn http://default:logline@127.0.0.1:8123 'index=app sourcetype=json | table _time host msg'

Known gaps#

• Vector's HEC /services/collector/raw endpoint ignores the host, index, source and sourcetype query parameters that Splunk accepts; events get the input's defaults. The /event endpoint honors the metadata in the payload. The engine will front HEC itself later.
• ClickHouse infers DateTime64 for string values that look like timestamps when it inserts into the fields JSON column, so a ts field reads back as 2026-11-05 14:28:00.250000000 rather than the original string. The installer's ClickHouse profile will set input_format_try_infer_datetimes=0 and input_format_try_infer_dates=0; the throwaway dev container does not.
• inputs and sourcetypes are files edited by the CLI. The web UI and the SQLite store come with the engine's HTTP server.