[FEATURE] PPL Asynchronous Query API

Start by reviewing the existing synchronous PPL endpoint at POST /_plugins/_ppl and its current response envelope. Trace how submit, poll, and cancellation would map to the documented job lifecycle, pagination, sequencing, leases, and update modes; done means the documented async endpoints and status semantics are implemented consistently.

Overview

Add an asynchronous lifecycle API for Calcite PPL queries while preserving the existing synchronous POST /_plugins/_ppl behavior.

A client can submit a query, poll its current state and complete result, and cancel or delete it. Every submit or poll response is a complete current snapshot. This scope does not expose partial results, progress, paging, windows, or delta delivery.

The current implementation relies on plugins.query.size_limit (10,000 rows by default). Supporting substantially larger results requires a separate pagination or delta-delivery design.

Endpoints

Purpose	Method	Path	Authorization action
Submit or synchronous execute	POST	/_plugins/_ppl	cluster:admin/opensearch/ppl
Poll and renew lease	GET	/_plugins/_ppl/jobs/{id}	cluster:admin/opensearch/ppl/async_query/result
Cancel/delete	DELETE	/_plugins/_ppl/jobs/{id}	cluster:admin/opensearch/ppl/async_query/delete

Submit

Presence of either wait_for_completion_timeout or keep_alive selects asynchronous execution. If both fields are absent, the existing synchronous path and response remain unchanged.

{
  "query": "source=account | stats count() by age",
  "wait_for_completion_timeout": "5s",
  "keep_alive": "5m"
}

Field	Required	Default	Limit
query	yes	none	Valid Calcite PPL
wait_for_completion_timeout	no	5s	0s to 60s
keep_alive	no	5m	Greater than 0, at most 24h

If the query succeeds or fails within the wait period, submit returns the terminal response directly without an ID and does not retain state.

If the query is still running when the wait period ends:

{
  "id": "<opaque-id>",
  "status": "RUNNING",
  "schema": [],
  "datarows": [],
  "total": 0
}

Poll

GET /_plugins/_ppl/jobs/{id}?keep_alive=5m

An authorized GET renews the lease relative to the current time. If keep_alive is omitted, the job's current lease interval is reused.

A successful terminal response contains the same schema, rows, and defined ordering as the equivalent synchronous PPL query:

{
  "id": "<opaque-id>",
  "status": "SUCCEEDED",
  "took": 214,
  "schema": [
    {"name": "count()", "type": "long"},
    {"name": "age", "type": "integer"}
  ],
  "datarows": [
    [521, 20],
    [442, 30],
    [398, 40]
  ],
  "total": 3
}

Every response is the complete current snapshot. total equals datarows.length.

Lifecycle behavior

State/event	HTTP	Behavior
Running at submit timeout	200	Return ID, RUNNING, and an empty result
Fast success	200	Return final result without ID; do not retain
Success after submit timeout	200 on GET	Return and retain full final result
Execution failure	200	Return FAILED plus a sanitized error and no rows
DELETE of running job	200	Return CANCELLED, cancel execution, remove state
DELETE of terminal job	200	Return prior terminal status and remove state
Unknown, expired, deleted, or owner-node-lost	404	Return generic not found
Unauthorized action or job visibility	403	Do not disclose metadata
Capacity exhausted	429	Reject before execution

Only RUNNING, SUCCEEDED, and FAILED are retained and pollable. CANCELLED is a DELETE acknowledgement, not a retained state.

Multi-node behavior

The opaque ID contains owner-routing information but no query, user, index, timing, or result data. GET and DELETE can reach any cluster node and are routed to the owner. State is node-local and non-durable; owner departure or restart results in 404.

Security

• Submit, GET, and DELETE are independently authorized operations.
• Submission requires the same PPL and source-index permissions as the equivalent synchronous query.
• Every GET and DELETE request is reauthorized through OpenSearch FGAC, including requests forwarded to the owner node.
• Job visibility requires the same principal and requested tenant, and the caller's current backend roles must contain the roles captured at submission.
• Knowing an opaque job ID grants no access.
• Asynchronous code must not write query text, job IDs, user identities, schema, rows, or error payloads to unprotected logs or metric dimensions.
• Asynchronous responses use Cache-Control: no-store.

Resource limits

Setting	Default
plugins.ppl.async.node_concurrent_running_queries	20
plugins.ppl.async.max_retained_jobs	100
plugins.ppl.async.max_wait_for_completion_timeout	60s
plugins.ppl.async.max_keep_alive	24h