Search jobs API

POST /api/search runs a search inside the request: the caller waits for the answer, a search over a concurrency cap is refused with 429, and a client that goes away kills the query. A search job is the other shape: submit it, poll it or wait on it, read its result, cancel it. Jobs are what scripts and other programs should use, and they live under /api/v1, the versioned API the engine describes in OpenAPI 3.1 at GET /api/v1/openapi.json. The unversioned /api endpoints are the web UI's and may change with it.

A job's life#

POST /api/v1/jobs                      submit        -> 202, Location: /api/v1/jobs/{id}
GET  /api/v1/jobs/{id}?wait=10         poll or wait  -> state: queued | running | done | failed | cancelled
GET  /api/v1/jobs/{id}/results         read          -> the search response, paged, or CSV
POST /api/v1/jobs/{id}/search          slice         -> commands run over the result in the engine
DELETE /api/v1/jobs/{id}               cancel, forget

A job is submitted with the same body as POST /api/search (query, earliest, latest, limit, tz, fields, timeline) plus ttl_s, how long to keep the result once it finishes, and wait_s, which makes the submit wait up to that many seconds for the job to finish before answering, so a short search needs one call:

curl -s -H "Authorization: Bearer ll_..." -H "Content-Type: application/json" \
  https://host/api/v1/jobs -d '{"query":"index=web | stats count by status","earliest":"-1h","wait_s":10}'

{"job":{"id":"3fd50ce7b1406f4a","state":"done","user":"ci","via":"token",
 "query":"index=web | stats count by status","earliest":"-1h","latest":"now","limit":1000,
 "fields":false,"timeline":false,"ttl_s":600,
 "created":"2026-11-07T16:07:21Z","started":"2026-11-07T16:07:21Z","finished":"2026-11-07T16:07:21Z","expires":"2026-11-07T16:17:21Z",
 "queue_ahead":0,
 "result":{"events":false,"columns":["status","count"],"count":5,"limited":false,"elapsed_ms":41.2},
 "results":"/api/v1/jobs/3fd50ce7b1406f4a/results"}}

The answer is 200 when the job finished within wait_s, else 202 with the job as it is. GET /api/v1/jobs/{id} returns the same document, and ?wait=N (at most 30 seconds) waits on it the same way. A job's state:

GET /api/v1/jobs/{id}/results returns the search response as POST /api/search does (columns, rows, sql, limited, the histogram and field summary when asked for), with offset and count paging the rows (count=0 means all from offset on) and total the whole result's rows; format=csv returns the page as a CSV file. Before the job is done it answers 409 with the state (queued, running, cancelled) and the job; a failed job's results answer with its own error, status and body.

DELETE /api/v1/jobs/{id} drops a queued job, cancels a running one's search (its ClickHouse query is killed) and forgets a finished one's result. GET /api/v1/jobs lists the caller's jobs newest first, and the counts of queued, running and kept jobs host-wide; an admin's list has every account's. A job is visible to the account that submitted it and to admins; anyone else's reads as 404, so ids do not leak.

Searching a result#

POST /api/v1/jobs/{id}/search runs a pipeline of commands over a done job's result in the engine, without another ClickHouse query: the job's rows are the input, and the answer is a search response (columns, rows, count, limited, residual, and the range and sql of the job's own search). The body is the commands, starting with a pipe:

curl -s -H "Authorization: Bearer ll_..." -H "Content-Type: application/json" \
  https://host/api/v1/jobs/3fd50ce7b1406f4a/search -d '{"query":"| where count > 10 | stats sum(count) AS total"}'

Every command with an in-process form is allowed (COMPATIBILITY.md, "In-process execution"); one without answers 400 (kind plan), a query that does not parse 400 (kind parse), and a job that is not done answers as its results would. The job's own result is unchanged, so one job can be sliced many times: this is how a dashboard runs a base search once and gives each panel its own view of it (DASHBOARDS.md).

A slice sees what the job's caps let through. A result that ends in a transforming command (stats, timechart) is complete; raw events are at most limit of them, newest first, and when limited is true a slice aggregates only those. The engine's rows cap applies to a slice's output as it does to a search's.

Queueing and limits#

A job runs under the same caps as a request's search (SEARCH-LIMITS.md): the host's concurrency cap and the account's own. Where a request over a cap is refused, a job waits. Jobs start in submission order; one whose account's cap is full is passed over, so one account's backlog does not hold the others up, and tried again when a slot frees. A running job is a running search: it shows in GET /api/searches and on the Health page, its query id is the job id, and DELETE /api/searches/{id} cancels it as it would any search. It runs as the account that submitted it, with that account's event, row and byte caps, and the search profile's limits apply to its query.

A finished job keeps its result for ttl_s seconds (default 600, at most 3600) and is then dropped; results live in the engine's memory and are bounded by the result caps, so a job never holds more than a request could return. An account holds at most 50 jobs at once in any state: over that, its oldest finished jobs are dropped to make room, and when every held job is still queued or running the submit is refused:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

{"cap":50,"error":"too many search jobs for ci: 50 of 50 queued or running; wait for one to finish, or cancel or delete one",
 "held":50,"kind":"busy","queue_scope":"user","retry_after_s":5,"scope":"queue"}

The host holds at most 1,000 jobs in all, the same way (queue_scope global). Jobs do not survive an engine restart: on shutdown every queued and running job is cancelled and its query killed, and nothing is kept.

Authentication and audit#

Every endpoint takes a session or an API token, search-scoped tokens included; a session must send X-Logline-Client on the submit and the delete, as on every state-changing request (AUTH.md). A readonly account may run jobs, as it may search.

index=_audit event=search details.job=* | stats avg(details.elapsed_ms), count by actor

Not yet#

• A jobs view in the UI, and the search page running on jobs rather than the synchronous endpoint (the dashboards run their shared base searches as jobs already).
• Jobs that outlive the engine: results kept on disk or in ClickHouse, with a job history to search.
• Progress while a job runs (rows read, bytes scanned) from ClickHouse's system.processes.
• Saved searches and alerts on top of jobs (phase 1b).
• Streaming a large result rather than paging a result held in memory.
• A Splunk-shaped jobs endpoint for Splunk SDKs and scripts, pending counsel (DESIGN.md, "Questions for counsel" 3).