Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[BUG] Simple `top` queries on high-cardinality fields are very slow

Abierto
  • ¿Algo no está bien?
#5,673 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
java, sql
Área
backend-api-design, performance

Línea de trabajo

The reported entry point is a PPL top query; start with the ppl --explain logical and physical plans and their aggregation pushdown, especially the generated composite_buckets request. Compare it with the shown terms/top_values request; done means trivial top queries avoid the coordinator scan and return the expected top results.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug calcite performance PPL pushdown

What is the bug?
Simple top queries can take more than 30s if done on fields with high cardinality (>1mil distinct values). This happens even if there's a small top limit such as the default top 10.

How can one reproduce the bug?
Steps to reproduce the behavior:

  1. Index data with lots of distinct request_id values (>1mil, keyword-typed)
  2. source=idx | top request_id
  3. Measure

Explain plan:

> echo 'source=logs | top request_id' | ppl --explain
= Calcite Plan =
== Logical ==
LogicalSystemLimit(fetch=[10000], type=[QUERY_SIZE_LIMIT])
  LogicalProject(request_id=[$0], count=[$1])
    LogicalFilter(condition=[=($2, 10)])
      LogicalProject(request_id=[$0], count=[$1], _row_number_rare_top_=[ROW_NUMBER() OVER (ORDER BY $1 DESC, $0)])
        LogicalAggregate(group=[{0}], count=[COUNT()])
          LogicalProject(request_id=[$60])
            CalciteLogicalIndexScan(table=[[OpenSearch, logs]])


== Physical ==
EnumerableLimit(fetch=[10000])
  EnumerableCalc(expr#0..2=[{inputs}], expr#3=[10], expr#4=[=($t2, $t3)], proj#0..1=[{exprs}], $condition=[$t4])
    EnumerableWindow(window#0=[window(order by [1 DESC, 0] rows between UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])])
      CalciteEnumerableIndexScan(table=[[OpenSearch, logs]], PushDownContext=[[AGGREGATION-rel#50:LogicalAggregate.NONE.[](input=RelSubset#49,group={0},count=COUNT())], OpenSearchRequestBuilder(sourceBuilder={
        "aggregations": {
          "composite_buckets": {
            "composite": {
              "size": 10000,
              "sources": [
                {
                  "request_id": {
                    "terms": {
                      "field": "request_id",
                      "missing_bucket": true,
                      "missing_order": "first",
                      "order": "asc"
                    }
                  }
                }
              ]
            }
          }
        },
        "from": 0,
        "size": 0,
        "timeout": "1m"
      }, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

The inner DSL in this case takes ~300ms (index cache disabled), while the PPL takes ~43800ms on my machine. However, it's also ordering keys lexically instead of by count using OpenSearch's own top_values aggregation.

What is the expected behavior?

At least this case of a trivial top aggregation (without trailing consuming commands) should be fast. OpenSearch itself can do this quickly (133 ms) with straightforward DSL, including returning an accurate size for the "other" bucket (sum_other_doc_count):

{
  "size": 0,
  "aggs": {
    "top_values": {
      "terms": {
        "field": "request_id",
        "size": 10
      }
    }
  }
}

{
    "took": 99,
    "timed_out": false,
    "terminated_early": true,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 10000,
            "relation": "gte"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "top_values": {
            "doc_count_error_upper_bound": 4,
            "sum_other_doc_count": 999990,
            "buckets": [
                {
                    "key": "00000067-142d-4d90-8a2c-28409600adbd",
                    "doc_count": 1
                },
                {
                    "key": "00001910-7d05-4e6a-9716-2c74137475e6",
                    "doc_count": 1
                },
                {
                    "key": "00002676-4f8c-428a-a29a-b862ca672cb7",
                    "doc_count": 1
                },
                {
                    "key": "00003943-8463-46f2-a92b-641b4afbafb1",
                    "doc_count": 1
                },
                {
                    "key": "00003f95-b049-41de-b9be-03aae071ad99",
                    "doc_count": 1
                },
                {
                    "key": "00005502-587d-4b00-865c-86f2af5502c2",
                    "doc_count": 1
                },
                {
                    "key": "00005754-ad22-4894-a9b5-dba3007ff19a",
                    "doc_count": 1
                },
                {
                    "key": "00005c50-97d3-4c11-ad74-90a3219ca815",
                    "doc_count": 1
                },
                {
                    "key": "000066c7-0301-45aa-98b1-d4997a590214",
                    "doc_count": 1
                },
                {
                    "key": "00007501-890e-4f0b-a688-2653ade97f8b",
                    "doc_count": 1
                }
            ]
        }
    }
}

What is your host/environment?

  • mainline, also tested 3.5
  • linux (AL2023), jdk 25

Do you have any screenshots?
N/A

Do you have any additional context?
Seems to be that we don't have any top_values aggregation pushdown at all, so this query turns into a coordinator scan.

Lenguaje dominante
Java
Estrellas
175
Forks
231
Merge medio
2 d 7 h
PR fusionados (30 d)
35

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de opensearch-project/sql

Todos los issues de opensearch-project/sql

Issues similares

Más issues de Java

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.