Search limits

One bad search must not take the host down. Every search runs under limits on time, memory, how much it reads and returns, and how many searches run at once, enforced in two places that agree with each other:

• ClickHouse runs searches as a dedicated account, logline_search, whose settings profile bounds each query. Constraints pin those bounds, so no query can change them, not even one a planner bug produced.
• The engine caps how many searches run, cuts results that are too large and says so, gives up on a search a little after ClickHouse should have, and then kills the query in ClickHouse by its id.

The installer derives every limit from the host's RAM, in the same function that sizes ClickHouse and Vector (DESIGN.md 8.1), and an admin can override any of them.

What bounds a search#

Defaults per resource profile (small up to 6 GiB of RAM, medium up to 24 GiB, large above). The memory and thread figures follow the host; the examples are for 4 GiB and 2 CPUs, 16 GiB and 8 CPUs, 64 GiB and 32 CPUs.

ClickHouse's own result limits, max_result_rows (twice the larger of the event and table-row caps, plus one) and max_result_bytes (four times the byte cap), sit above the engine's and throw rather than cut. A normal search never reaches them, because the engine asks for one row more than its cap and cuts there; they are a backstop for a planner bug. ClickHouse applies them to the final result only, not to the subqueries every search compiles to, which was checked on ClickHouse 26.8.

The time limit does not grow with the host, since a bigger host finishes the same search sooner. The read limits are backstops behind it, sized to roughly what such a host scans in that time.

The search account#

users.xml (written by logline install) has two accounts, both reachable from loopback only:

• logline: ingest, the schema, backups, health, and KILL QUERY.
• logline_search: searches only. It may SELECT from the logline database and nothing else, under the logline_search profile: readonly=2, the limits above, result_overflow_mode, timeout_overflow_mode and read_overflow_mode set to throw, and cancel_http_readonly_queries_on_client_close=1.

readonly=2 allows reading and changing settings; readonly=1 would also refuse the SETTINGS clause spath compiles to and the output-format settings the engine sends with every query. Every limit and overflow mode has a const constraint, so a query that tries to change one, up or down, fails with SETTING_CONSTRAINT_VIOLATION.

The engine reads the account from LOGLINE_CH_SEARCH_USER and LOGLINE_CH_SEARCH_PASSWORD in engine.env, and logline spl run does the same. Without them, as in development, searches run as the main account without the profile, and the engine logs a warning at start.

Cancellation#

Every search has a ClickHouse query_id: the request id (the X-Request-Id response header and the request_id in the logs), with -timeline for the histogram's query. When the engine gives up on a search, it runs KILL QUERY WHERE query_id IN (...) SYNC as the main account, which waits until ClickHouse has stopped it. It gives up when:

• its time limit passes, a little after ClickHouse's (504, kind timeout);
• someone cancels it with DELETE /api/searches/{id} (409, kind cancelled);
• the client goes away (the browser tab closes, curl gives up); nobody reads the answer, and the request log shows 499.

Without the kill, ClickHouse goes on running a query nobody will read: on ClickHouse 26.8 a query whose client had given up was still running two seconds later, with cancel_http_readonly_queries_on_client_close set. logline spl run kills its query on Ctrl-C too.

Concurrency#

The engine counts running searches against the host's cap and the account's: PER_USER, or PER_ADMIN for an admin. An API token's searches count against the account that owns it. Over a cap the search is refused at once with 429, Retry-After: 5 and kind busy, naming the cap:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

{"cap":2,"error":"too many searches running for alice: 2 of 2; wait for one to finish or cancel it",
 "kind":"busy","retry_after_s":5,"running":2,"scope":"user"}

scope is user for the account's cap and global for the host's. The web UI shows the message and when to retry.

A refused search does not wait in a queue. A search job does: it takes a slot under the same caps when one frees, in submission order (JOBS.md), and while it runs it is one of the searches here.

Results that are too large#

A search returns at most the event cap for raw events, and for a results table at most the table-row cap. Any result is also cut once what the engine has read from ClickHouse passes the byte cap. A cut result says so: limited is true and limited_by is events, rows or bytes. The caps apply to the rows ClickHouse returns, before any in-process step such as the timechart pivot; a results table an in-process command makes from them is then capped at the table-row cap too.

When ClickHouse stops a search#

A search that goes past one of the profile's limits fails with kind limit and the limit's name:

Changing the limits#

engine.env is rewritten by every install, so overrides go in /etc/logline/engine.local.env. The installer writes that file once, with every variable commented out and set to this host's derived value, and never rewrites it. Uncomment and change a line, then:

sudo logline install

The installer applies the overrides on top of the derived limits and writes the result to engine.env and to the logline_search profile in users.xml, then restarts the services. Both sides always come from the same values. A value that is not a positive whole number stops the install, naming the file and the variable.

logline doctor and the Health page show the limits in effect in the search limits check, read from the engine's environment, and compare them with what ClickHouse applies, read back as the search account. A mismatch (say, after a hand edit of engine.env) is a warning, and re-running the installer fixes it.

What is recorded#

index=_audit event=search.rejected | stats count by actor, details.scope
index=_audit event=search details.stopped=* | stats count by details.stopped, details.killed
index=_audit event=search details.clickhouse_limit=* | stats count by details.clickhouse_limit
index=_internal sourcetype="logline:engine" msg="search stopped" | table _time, request_id, reason, killed

The Health page shows the searches running now against the caps, with a Cancel button for each (HEALTH.md).

Not yet#

• logline tune to apply changed limits without a full re-run of the installer, and recomputing them when the host's RAM changes.
• Per-role or per-account limits beyond user and admin, and quotas over time (searches or rows read per hour); ClickHouse quotas can carry them.
• Fair sharing of slots between accounts while they wait: the job queue is first come, first served, skipping an account whose own cap is full.
• Limits on the CLI's direct ClickHouse access beyond the search profile: logline spl run has no concurrency cap.
• max_memory_usage_for_user across all running searches; today the server-wide max_server_memory_usage is that backstop.